SIMA
Free while signed in. Answers cite the passages they came from.
DeepMind's Scalable Instructable Multiworld Agent (SIMA) is a generalist AI agent that follows natural-language instructions across nine commercial 3D video games like No Man's Sky, Teardown, Valheim, and Space Engineers.
600-skill taxonomy: Evaluation covers 600 basic skills spanning navigation, object interaction, and menu use, giving a detailed picture of what generalist 3D agents can and can't do.
Training data: Built by recording human players instructing each other in gameplay and fine-tuning a pretrained vision-language backbone on this instruction-paired footage.
Cross-game generalization: Training across many games produces an agent that performs nearly as well on unseen games as on games it trained on - a strong generalization signal for 3D embodied agents.
Language matters: Ablations show that textual instruction is the dominant performance driver; removing the language conditioning collapses skill execution, underscoring language as an alignment interface for embodied agents.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack