V-JEPA 2
Free while signed in. Answers cite the passages they came from.
Meta AI introduces V-JEPA 2, a scalable joint-embedding predictive architecture for self-supervised video learning, targeting the goal of building a generalist world model capable of understanding, predicting, and planning in the physical world. The model is trained in two stages: action-free pretraining on 1M+ hours of internet videos and images, followed by post-training with only 62 hours of unlabeled robot trajectories (Droid dataset). This yields a latent video representation that is useful across a broad range of downstream tasks.
Video understanding & QA: V-JEPA 2 achieves state-of-the-art performance on motion-centric tasks like Something-Something v2 (77.3 top-1 acc) and Epic-Kitchens-100 action anticipation (39.7 recall@5), and excels in multimodal QA when aligned with an 8B language model (e.g., 84.0 on PerceptionTest, 76.9 on TempCompass). Notably, this is achieved without language supervision during video pretraining.
Self-supervised robot planning: With just 62 hours of robot video data, the team fine-tunes an action-conditioned model (V-JEPA 2-AC) on top of the frozen video encoder. This model supports zero-shot deployment for prehensile tasks like grasping and pick-and-place on real Franka arms in unseen environments, without rewards, demonstrations, or lab-specific fine-tuning.
Architectural scale-up: V-JEPA 2 scales beyond its predecessor using four key improvements: expanded dataset (22M videos), larger ViT-g model (1B params), longer training (up to 252k iterations), and progressive spatiotemporal resolution. Combined, these yield significant gains across visual benchmarks (e.g., +4.0 average accuracy gain over baseline).
Comparison to other models: V-JEPA 2 outperforms image encoders like DINOv2 and PEcoreG on video QA, and is competitive on appearance tasks like ImageNet. When used in MLLMs, it surpasses models like PLM-8B on PerceptionTest, MVP, and TOMATO using only vision encoders pretrained without language data.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack