Multimodal

AudioPaLM

First page

Paper summary

Fuses PaLM-2 and AudioLM into a multimodal architecture supporting speech understanding and generation.

Ask this paper

Key points

Unified speech-text: Represents both speech and text as tokens in a shared vocabulary, enabling any-to-any conversion between modalities.

Zero-shot translation: Performs zero-shot speech-to-text translation into languages never seen as translation targets during training.

Speech generation: Generates high-quality speech in the voice of the input speaker while preserving prosody.

Unified speech foundation: A precursor to 2024's fully multimodal systems like GPT-4o that natively process and generate speech.

Every Monday