NVIDIA paper fixes tool calls in duplex speech models
Developers building voice agents for customer service can apply this architecture to close the 34-point gap between speech and text agents without degrading conversation quality.
NVIDIA researchers show commercial duplex voice models complete only 31 to 51 percent of grounded customer-service tasks, while text agents like GPT-5 reach 85 percent on the same tasks. The performance loss occurs in the speech pipeline. Their fix routes decisions out of the speech model by emitting a delegation token to a text backend LLM. This architecture achieves 92.0 to 97.2 percent tool-call recall and 81.2 percent accuracy at rejecting irrelevant calls, while maintaining the baseline turn-taking rate and streaming ASR word error rate.
NVIDIA research papers are on fire recently! Here is another interesting paper where they give full-duplex speech models tool calls. (bookmark it) Commercial duplex voice models complete 31 to 51 percent of grounded customer-service tasks under clean conditions. Text agents like GPT-5 reach 85 percent on the same tasks in text mode. Most of what a voice agent loses, it loses in the speech pipeline. The fix routes the decision out of the speech model. The duplex frontend learns to emit a delegation token, forwards streaming transcripts to a text backend LLM for the tool call, and receives the result through a lightweight prefill-and-repeat mechanism before streaming TTS speaks it. Tool-call recall runs 92.0 to 97.2 percent with 81.2 percent accuracy at rejecting irrelevant calls. Turn-taking rate, streaming ASR word error rate and spoken-language intelligence all stay at the no-tool-call baseline. Paper: arxiv.org/abs/2609.19334 Chat with Paper: academy.dair.ai/papers/a-front… 💬 6 🔄 1 ❤️ 9 👀 1880 📊 9 ⚡
