Conversational Speech Model
Researchers from Sesame propose an end-to-end multimodal TTS approach for natural, context-aware speech in real-time conversational AI systems.
Ask this paper
Beyond one-to-many TTS - Traditional text-to-speech lacks rich contextual awareness. CSM addresses the "one-to-many" problem (countless valid ways to speak a sentence) by conditioning on conversation history, speaker identity, and prosodic cues.
End-to-end architecture on RVQ tokens - CSM directly models Residual Vector Quantization (RVQ) audio tokens via two autoregressive transformers: (1) a multimodal backbone that interleaves text/audio to generate the zeroth codebook level and (2) a lightweight decoder for the remaining codebooks. This single-stage design enhances efficiency and expressivity.
Compute amortization - Training on full RVQ codebooks is memory-heavy; to mitigate this, CSM only trains the decoder on a random 1/16 of frames while still learning the zeroth codebook fully. This preserves fidelity yet reduces computational load.
Strong evaluations -
Open-source and future plans - The team will release their models under Apache 2.0. Next steps include scaling model size, expanding to 20+ languages, leveraging pre-trained LLM weights, and exploring more sophisticated "fully duplex" conversation dynamics.