Code World Model
Free while signed in. Answers cite the passages they came from.
Meta FAIR releases CWM, a 32B open-weights coder trained to model code execution and to act inside containers. It mid-trains on Python interpreter traces and agentic Docker trajectories, then upgrades with multi-turn RL across SWE, coding, and math. CWM is both a strong coder and a testbed for world-model-style reasoning in software environments.
Execution-aware training recipe: Pretrain 8T tokens, then mid-train 5T on Python execution traces and ForagerAgent trajectories collected in containerized repos, followed by SFT (100B) and joint multi-task RL with a GRPO-style algorithm and asynchronous rollouts. Results include 120M traced functions, ~70k repo-level traces, and 3M agentic trajectories.
Model + context scaling: Dense 32B decoder with alternating local/global sliding-window attention and 131k max context. Scaled RoPE, GQA, FP8 training, and long-context bucketization are used to keep throughput sane. Inference can fit on a single 80 GB H100 with quantization.
Agentic RL design for SWE: The agent works inside a repo sandbox with a minimal toolset (bash, edit, create, submit), runs tests, builds patches with git diff, and is rewarded by hidden tests plus patch-similarity shaping. Self-bootstrapped traces improve format adherence before RL.
Performance highlights: On SWE-bench Verified, 53.9% base pass@1 and 65.8% with test-time scaling (best@k); chart on page 3 shows CWM competitive with much larger or closed models. Also LCB-v5 68.6, Math-500 96.6, AIME-24 76.0, CruxEval-Output 94.3.
Why it matters for AI devs: CWM exposes trace-prediction tokens to simulate Python execution in prompts, enabling grounded reasoning, neural-debugger workflows, and trace-guided code synthesis. Ablations show execution traces boost CruxEval, and ForagerAgent boosts agentic NLLs and SWE pass@1.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack