A frontend-backend architecture for tool calls in full-duplex speech models
Ke Hu and colleagues at NVIDIA give a full-duplex speech-to-speech model tool-calling ability by having the speech frontend emit a delegation token and hand streaming transcripts to a text backend LLM, rather than teaching the duplex model to call tools itself.
Ask this paper
Delegation token as the interface. The duplex speech-to-text frontend learns when to delegate, forwards ASR transcripts to a text backend, and receives results through a prefill-and-repeat mechanism before streaming TTS speaks the reply.
The gap this closes is large. Leading commercial duplex voice models complete only 31 to 51 percent of grounded customer-service tasks under clean conditions, while text agents such as GPT-5 reach 85 percent on the same tasks in text mode.
Tool-call recall of 92 to 97 percent. On BFCL the frontend recalls 97.2, 92.0, 95.0 and 93.5 percent on Simple, Multiple, Parallel and Parallel-Multiple, with 81.2 percent accuracy at rejecting irrelevant calls.
Conversational behavior is nearly unchanged. Turn-taking, streaming ASR word error rate and spoken-language intelligence stay close to the no-tool-call baseline, so delegation training does not degrade the duplex behavior.
Backend size carries the task completion. With a Qwen3-235B-A22B backend the system beats GPT-realtime-mini and Qwen3-Omni-30B-A3B-Instruct on EVA-Bench and reaches 72 percent airline-domain completion against GPT-realtime2.
Abstract
Full-duplex speech-to-speech (S2S) models provide natural, low-latency conversational interaction and would benefit from the ability to use external tools and complete voice-agent tasks. We propose a frontend-backend architecture where a duplex speech-to-text frontend learns to emit a delegation token and forwards streaming ASR transcripts to a text-based backend LLM for tool calls. Tool-call results from the backend are injected back into the frontend through a lightweight prefill-and-repeat mechanism and then synthesized using streaming TTS to the user. Our approach largely preserves regular duplex turn-taking, interruption handling, and low-latency interaction as it requires minimal modifications to the frontend model. In a single-turn tool-call evaluation, our system achieves 92-97% tool-call recall, competitive tool-call prediction performance, and 81.2% accuracy in rejecting irrelevant calls. When equipped with a larger backend (e.g., Qwen3-235B-A22B), our system achieves competitive results on Full-Duplex-Bench-V3 compared to open and closed source models, and significantly outperforms GPT-realtime-mini and Qwen3-Omni-30B-A3B-Instruct on EVA-Bench. These results demonstrate that backend delegation is an effective and modular approach for combining natural duplex speech interaction with strong agentic tool-call capabilities.