Rethinking JEPA
Free while signed in. Answers cite the passages they came from.
Apple proposes SALT (Static-teacher Asymmetric Latent Training), a simple 2-stage V-JEPA alternative that first trains a teacher with pixel reconstruction, then freezes it and trains a student to predict the teacher’s latents on masked regions. It removes EMA, decouples teacher and student, and gives a cleaner model selection while being more compute-efficient.
Recipe that scales without EMA: Stage 1: train a video encoder with a VideoMAE-style pixel reconstruction objective but using V-JEPA’s multi-block masking (called V-Pixel). Stage 2: freeze that encoder and train a student encoder+predictor to match the teacher’s latents on masked regions. Both losses are proper and stable, eliminating the collapse machinery.
Better frozen-backbone results at lower compute: At matched pretraining steps on the V-3.6M mix, SALT improves average Top-1 over V-JEPA 2 and scales well with student size. The ViT-g/G SALT students top SSv2 and are competitive on K400.
Weak teacher, strong student: Students trained by small or sub-optimal teachers still become SOTA-level. The best ViT-L student uses only a ViT-L teacher, and even a ViT-G student peaks with a ViT-L teacher.
An actually useful training signal: Unlike EMA JEPA, where loss is a poor proxy, SALT’s student training loss correlates tightly with downstream frozen accuracy, enabling interpretable model selection during pretraining.
Masking and data choices that matter: For the teacher, multi-block masking beats random tubes and causal masking. The data mix is robust: K710-only or Panda2.8M-only teachers still yield strong students, with V-3.6M best overall.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack