Conversational Speech Model
Free while signed in. Answers cite the passages they came from.
Researchers from Sesame propose an end-to-end multimodal TTS approach for natural, context-aware speech in real-time conversational AI systems.
Beyond one-to-many TTS - Traditional text-to-speech lacks rich contextual awareness. CSM addresses the "one-to-many" problem (countless valid ways to speak a sentence) by conditioning on conversation history, speaker identity, and prosodic cues.
End-to-end architecture on RVQ tokens - CSM directly models Residual Vector Quantization (RVQ) audio tokens via two autoregressive transformers: (1) a multimodal backbone that interleaves text/audio to generate the zeroth codebook level and (2) a lightweight decoder for the remaining codebooks. This single-stage design enhances efficiency and expressivity.
Compute amortization - Training on full RVQ codebooks is memory-heavy; to mitigate this, CSM only trains the decoder on a random 1/16 of frames while still learning the zeroth codebook fully. This preserves fidelity yet reduces computational load.
Strong evaluations -
Open-source and future plans - The team will release their models under Apache 2.0. Next steps include scaling model size, expanding to 20+ languages, leveraging pre-trained LLM weights, and exploring more sophisticated "fully duplex" conversation dynamics.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack