Language Models that Think, Chat Better
Free while signed in. Answers cite the passages they came from.
A simple recipe, RL with Model-rewarded Thinking, makes small open models “plan first, answer second” on regular chat prompts and trains them with online RL against a preference reward. On Llama-3.1-8B and Qwen-2.5-7B, this consistently beats standard RLHF on chat, creative writing, and general knowledge, with the best 8B model topping some frontier systems on WildBench and AlpacaEval2.
What’s new: Instead of rule-verifiable rewards (math, code), RLMT uses long chain-of-thought on diverse real-world prompts plus a reward model (Skywork) to score outputs, trained with online RL (GRPO, PPO, DPO).
Setup: Warm-start with small SFT on teacher-generated think→respond traces, then optimize with GRPO on ~7.5k WildChat-IF prompts. A “Zero” variant skips SFT and still works by prompting base models to emit think tags before answers.
Results at a glance: RLMT lifts chat scores by roughly 3–8 points over matched RLHF baselines. Table 1 reports Llama-3.1-8B-Instruct-RLMT at 50.4 (WildBench), 58.7 (AlpacaEval2), 22.9 (ArenaHardV2), and 84.3 (CreativeWritingV3), outperforming much larger open models and beating GPT-4o on WildBench.
Base models without SFT: With GRPO, RLMT-Zero notably upgrades chat ability from weak baselines; Qwen-2.5-7B-RLMT-Zero surpasses its vendor Instruct model on average chat metrics.
Why it works (and what matters): Ablations show prompt mixture quality and reward-model strength are pivotal (WildChat-IF and Skywork-V2 win). Post-RL, models plan differently: fewer linear checklists, more constraint enumeration, theme grouping, and iterative refinement. CoT and responses lengthen over training.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack