M3-Agent
Free while signed in. Answers cite the passages they came from.
A framework for agents that watch and listen to long videos, build entity-centric memories, and use multi-turn reasoning to answer questions. M3-Agent stores both episodic details and distilled semantic knowledge in a multimodal memory graph, then learns a retrieval-reasoning policy with RL. The authors also introduce M3-Bench, a long-video QA benchmark with robot-view and web videos. Results show consistent gains over strong prompting baselines.
Entity-centric long-term memory. Builds a multimodal graph with nodes for text, faces, and voices, plus edges for relations and cross-modal identity links. Episodic entries record events, and semantic entries capture attributes and world knowledge. Conflicts are resolved by weight-based voting to keep memory consistent.
Online memorization with identity tools. Processes video streams clip by clip, using face recognition and speaker identification to maintain persistent character IDs, then writes grounded episodic and semantic memories keyed by those IDs.
RL-trained control for retrieval and reasoning. A policy model decides when to search memory and when to answer, performing iterative, multi-round queries over the memory store rather than single-shot RAG.
M3-Bench for long-video QA. 100 robot-perspective videos and 929 web videos with 6,313 total QA pairs targeting multi-detail, multi-hop, cross-modal, human understanding, and general knowledge questions.
State-of-the-art results and ablations. M3-Agent beats a Gemini-GPT-4o hybrid and other baselines on M3-Bench-robot, M3-Bench-web, and VideoMME-long. Semantic memory and identity equivalence are crucial, and RL training plus inter-turn instructions and explicit reasoning materially improve accuracy.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack