ACE-RL
Free while signed in. Answers cite the passages they came from.
A reinforcement-learning framework that replaces coarse, preference-pair rewards with instruction-specific, verifiable checklists. ACE-RL turns each long-form task into a set of explicit and implicit constraints, scores a model’s output by how well it satisfies them, and mixes this with a length-control reward during GRPO training. The result is stronger, more controllable long-form writing across domains and styles.
Key idea: Automatically deconstruct each instruction into a fine-grained checklist (explicit and implicit demands), then verify each item with a small LLM using a 3-level rubric (Fully/Partially/Not Met). Rewards = mean checklist score + a length reward, optimized with GRPO.
Why it matters: Moves beyond relevance/coherence/helpfulness toward instruction-adaptive quality. No preference pairs required, which lowers cost and improves scalability.
Data & setup: 32K long-form instructions, average 5.48 constraints per prompt, target length around 2.3K words. Verifier uses Qwen3-8B; length reward penalizes deviations beyond a tolerance band.
Results: On WritingBench, ACE-RL lifts models substantially over SFT and LLM-as-judge RL; e.g., Qwen-2.5-7B jumps from 57.0 to 78.6. A small Qwen-3-4B-thinking model trained with ACE-RL beats several proprietary and writing-tuned systems. On Arena-Write, win-rates reach ~68% vs six strong baselines.
Ablations & insights: Constraint-based rewards produce higher within-group reward variance than LLM-as-judge, indicating better discrimination among rollouts. Works with small reward models and even self-reward settings. Thinking mode plus ACE-RL outperforms non-thinking for long-form generation.
Constraint-based rewards produce higher within-group reward variance than LLM-as-judge, indicating better discrimination among rollouts.
Works with small reward models and even self-reward settings.
Thinking mode plus ACE-RL outperforms non-thinking for long-form generation.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack