ProPerAgent: Toward Memory-Driven Proactive Assistance
Shenglai Zeng, Xiao Yang, Wen-tau Yih, Xin Luna Dong and colleagues (Meta Reality Labs, Meta Superintelligence Labs, Michigan State) introduce ProPerMem, a benchmark for proactive assistance where the evidence for speaking up must be mined from a year of personal memory, and ProPerAgent, a harness that raises strict accuracy from under 10% to about 38%.
Ask this paper
Benchmark. ProPerMem pairs egocentric video with year-long, distractor-rich user memories averaging about 618K tokens, human-verified trigger windows, and scoring that checks both what the assistant says and when it says it.
Baselines fail. Every baseline, including frontier models with summarized or retrieval-augmented memory, scores at most 15%.
Controlled diagnosis. With oracle evidence delivered inside the trigger window, GPT-5.3-Chat reaches 96%. Letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are each necessary.
ProPerAgent. A plug-and-play harness surfaces memory from a graph-organized store and gates it to what is worth an interruption. Without training it lifts strict accuracy to 37.5% on Qwen3.8-27B and 38% on GPT-5.3-Chat. A light RL stage that penalizes early and repeated firing reaches 42.4%.
Why it matters. Proactive assistants on wearables need to decide when to speak, not only what to retrieve. This isolates timing as a separate failure that retrieval quality does not fix.
Abstract
As assistants move from reactive answering toward proactive intervention, memory changes role: it no longer only supplies the answer to a query, it determines whether and when to speak at all. Prior work in this setting hands the system the relevant fact in context; we study the regime that makes the problem hard, where the evidence justifying an intervention must be mined from a long-horizon personal memory. We introduce ProPerMem, a benchmark for memory-driven proactive assistance (MPA), pairing egocentric video with year-long, distractor-rich user memories (~618K tokens on average), human-verified trigger windows, and a protocol that scores content and timing jointly. Every baseline we evaluate performs poorly (at most 15%), including frontier models given summarized or retrieval-augmented memory. A controlled study explains why: with oracle evidence delivered inside the trigger window GPT-5.3-Chat reaches 96%; letting it choose the timing drops it to 38%, and replacing oracle evidence with retrieval drops it to 15%. Content and timing are separately necessary, and missing either is catastrophic. Guided by this, we build ProPerAgent (Proactive Personalized Agent) with a plug-and-play harness that surfaces memory from a graph-organized store and gates it down to what is worth interrupting for, plus a lightweight RL stage that curbs early and repeated firing. Without any training, ProPerAgent lifts strict accuracy from below 10% to 37.5% on Qwen3.8-27B and 38% on GPT-5.3-Chat; RL post-training reaches 42.4%.