GLM-4.5
Free while signed in. Answers cite the passages they came from.
An open Mixture‑of‑Experts family that targets a single model excelling across agentic tool use, complex reasoning, and real‑world coding. GLM‑4.5 (355B total, 32B active) introduces hybrid “thinking vs direct” modes, multi‑stage pretrain + mid‑train to 128K context, and extensive RL for reasoning, agents, and instruction following. It ranks near the top on a 12‑bench ARC suite and releases weights and eval tooling.
Results at a glance – On the 12‑benchmark ARC suite, GLM‑4.5 averages 3rd overall and 2nd on agentic tasks; Key scores: TAU‑Bench 70.1, BFCL‑V3 77.8, BrowseComp 26.4; AIME24 91.0, GPQA 79.1; SWE‑bench Verified 64.2, Terminal‑Bench 37.5.
Architecture and scaling choices – MoE with loss‑free balance routing and sigmoid gates, GQA with partial RoPE, QK‑Norm, 96 attention heads at 5,120 hidden, and an MoE Multi‑Token Prediction layer for speculative decoding.
Training recipe – 23T tokens pretrain with quality‑bucketed web, code, math, science, and multilingual data; mid‑training adds repo‑level code sequences, synthetic reasoning traces, 128K context, and large synthetic agent trajectories. Optimizer is Muon with cosine decay and sequence‑length extension plus RoPE base adjustment.
Post‑training and RL – Two‑stage expert‑then‑unified SFT + RL. Reasoning RL uses a difficulty curriculum, single‑stage 64K output‑length RL, token‑weighted loss for code, and strict filtering for science. Agentic RL covers web search and SWE with outcome rewards, strict format penalties, iterative self‑distillation, and turn‑scaling benefits. A new XML‑tagged function‑call template reduces escaping overhead.
RL infrastructure for agents – Slime provides synchronous training for general RL and decoupled asynchronous rollouts for long‑horizon agent tasks, with FP8 inference for faster data generation.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack