ATOKEN
Free while signed in. Answers cite the passages they came from.
ATOKEN introduces a single transformer tokenizer that works for images, videos, and 3D assets. It encodes all inputs into a shared sparse 4D latent space with 4D RoPE, trains without adversarial losses, and supports both continuous and discrete tokens. The paper reports strong reconstruction quality and solid semantic alignment, enabling both generation and understanding across modalities.
One latent space for 2D, video, and 3D. Inputs are patchified into sparse (t, x, y, z) features, so images are 2D slices, videos add time, and 3D uses surface voxels aggregated from multiview renders.
Pure Transformer with 4D RoPE and native resolution. The encoder extends a SigLIP2 vision tower to spaceātime blocks and adds 4D rotary positions, while the decoder mirrors the transformer to reconstruct pixels or 3D Gaussians. Native resolution and KV-cached temporal tiling speed video inference.
Adversarial-free training that targets texture statistics. Instead of GANs, the loss mixes L1, LPIPS, CLIP perceptual, and a Gram-matrix term, motivated by an rFID decomposition showing covariance dominates error.
Progressive curriculum across modalities. Four stages grow capability: image recon, add video, add 3D, then optional FSQ quantization.
Results across the board. With continuous latents, ATOKEN reports 0.21 rFID and 82.2% ImageNet zero-shot accuracy for images, 36.07 PSNR and 3.01 rFVD for video, and 28.28 PSNR with 90.9% 3D classification on Toys4k. Discrete FSQ tokens remain competitive while enabling AR generation and image-to-3D.
Get next weekās papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack