MMPostTrainBench: Benchmarking Autonomous Research for Multimodal Post-Training
Yuxin Liu, Zhendong Mao (USTC) and colleagues at Alibaba Group, Tsinghua and SJTU introduce MMPostTrainBench, which tests whether research agents can improve a multimodal model through post-training within a fixed budget, and propose the MMResearch framework to reduce the failures it finds.
Ask this paper
Benchmark. Eight tasks cover image, audio, video and joint audio-video understanding plus image-grounded software repair. Agents start from a common base model, use development feedback, and submit a model that is evaluated independently on target and non-target tasks.
Most runs make the model worse. Across six agent configurations, 52.1% of the 48 model-task means fall below the base model and the largest mean gain is 0.73 pp. GPT-5.6-terra uses 23.7 of its 24 hours and still ends 0.27 pp below base.
Regressions and selection. Gains on the target often cost other skills; one Qwen MMMU-Pro submission loses 9.88 pp on MMAR. Final submissions trail the agent's own best evaluated candidate by up to 5.38 pp.
MMResearch. Adding multimodal evidence reasoning, hierarchical memory and evaluation-guided model selection improves submitted accuracy by up to 7.75 pp for Claude Opus 4.8 with Claude Code and 2.33 pp for GPT-5.6-sol with Codex. Opus 5 and GPT-5.6-sol tie on mean score (44.42% vs 44.36%) with Opus using about 31% fewer tokens.
Abstract
Autonomous research seeks sustained model improvements through iterative experimentation and feedback. LLM agents show promise in automating machine learning and language-model post-training, but their ability to sustain multimodal improvement remains unclear. We introduce MMPostTrainBench, a benchmark spanning eight tasks in image, audio, video, and joint audio-video understanding and image-grounded software repair. Agents operate from a common base model within fixed budgets, using development feedback before independent evaluation of their submitted models. Evaluation covers target and non-target model outcomes, iterative model improvement and selection, and research integrity. Across all eight tasks, 52.1% of model--task means fall below the base, and evaluated submissions also exhibit non-target regressions. Model performance does not consistently improve across research iterations, and agents do not reliably select the best evaluated candidate for submission; final submissions trail that candidate by up to 5.38 percentage points. Extending autonomous research from text-only to multimodal tasks introduces additional sources of error in perception, cross-modal alignment, and temporal grounding. The observed regressions and selection gaps highlight the need to balance targeted improvements with non-target capability preservation and to retain gains across research iterations. These requirements motivate MMResearch, a multimodal research framework that connects media-grounded evidence to hypotheses and interventions, carries findings across rounds through hierarchical memory, and retains candidates using development evaluation. Added to existing code-agent runtimes, it improves submitted-model accuracy by up to 7.75 percentage points for Claude Opus 4.8 with Claude Code and 2.33 points for GPT-5.6-sol with Codex.