Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis
Sanyuan Chen and colleagues at FAIR at Meta present Text-AB, a 3B latent-diffusion speech model that removes the forced-alignment stage from the Audiobox line and handles dubbing and two-speaker dialogue in one system.
Ask this paper
Alignment-free text conditioning. Raw text goes through an off-the-shelf encoder and text-speech correspondence is learned through cross-attention, so no forced aligner and no explicit duration predictor is needed. This removes a preprocessing stage that has been standard in production TTS pipelines.
A much cheaper latent space. DAC-VAE features compress 48 kHz audio to a 25 Hz latent sequence, over 10x more compression than the EnCodec representation used previously, and resynthesis quality still improves. Higher compression at equal or better quality is what makes the 3B model affordable to train on 480k hours.
One pretrained model, three fine-tuned tasks. Cross-lingual dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue are all supervised fine-tunes of the same backbone, which is evidence that full-duplex conversation does not need a separately designed architecture.
Long-form generation by composition. One-shot generation covers about a minute; beyond that a multi-diffusion scheme stitches segments, and a multi-stage reranker picks among candidates using automated metrics. Reranking at inference is the same test-time-compute pattern now common in text models.
Why it matters for agent builders. Full-duplex dialogue synthesis is the output half of a real-time voice agent, and this is the first FAIR system to report it as a fine-tune of a general speech model rather than a bespoke stack.
Abstract
We present Alignment-Free Text-Audiobox (Text-AB), a unified framework for high-quality voice dubbing and full-duplex dialogue synthesis. Building on a Diffusion Transformer trained with a flow-matching objective, Text-AB departs from the Audiobox system along three dimensions. First, it operates in a latent diffusion framework using DAC-VAE features that encode 48 kHz waveforms into a 25 Hz latent sequence, giving over 10x higher compression than previous EnCodec representations while improving resynthesis quality. Second, Text-AB is alignment-free: it consumes raw text via an off-the-shelf text encoder and learns text-speech alignment through cross-attention, removing the need for forced alignment and explicit duration prediction. Third, we scale model and data substantially, pretraining a 3B-parameter model on 480k hours of monolingual speech, followed by supervised fine-tuning on three downstream tasks: cross-lingual voice dubbing, full-duplex dialogue synthesis, and emotional full-duplex dialogue synthesis. At inference, Text-AB supports one-shot generation for up to ~1 min of speech and arbitrarily long-form generation via a multi-diffusion scheme, plus a multi-stage reranking strategy that enhances quality based on automated metrics. On a real-world dubbing benchmark, Text-AB delivers a step-change improvement over the latest internal dubbing system, with large gains in prosody similarity, voice similarity, naturalness, and shareability. For full-duplex dialogue synthesis, it approaches human recordings on short-form conversations and substantially outperforms the latest internal model on long-form human-likeness and expressivity, while natively modeling turn-taking, back-channeling, and emotional dynamics. For emotional dialogue synthesis, emotion conditioning significantly improves emotion alignment and emotional interaction quality over the unconditioned baseline.