ElevenLabs Announces Fine-tuning, Interaction Engine and ElevenAI Assistant
The routing approach in the Interaction Engine — mixing cheap and frontier LLMs within one call — signals where voice agent economics are heading and gives enterprises a template for cutting token costs. Teams building customer-support or phone-based agents should evaluate the fine-tuning programme, since tailored speech models on proprietary data remain rare. The announcements also mark a direct rivalry with OpenAI's realtime voice stack and general work assistants like Granola or Wispr Flow.
At the ElevenLabs Summit in New York City, the voice AI company announced a set of upcoming products and features centred on voice agents. From the coming months, enterprises will be able to work with the ElevenLabs team to continually fine-tune its Text to Speech and Speech to Text models, along with open-source LLMs, on their own data.
The new Interaction Engine routes each moment of a conversation to the appropriate model: cheaper LLMs handle simple requests while frontier models step in for complex tasks such as processing refunds, reducing token spend without sacrificing outcomes. ElevenAgents gain Memory, letting agents retain information mentioned in passing so businesses can act on it, and Routines, which run operations before and after calls — for example pre-screening a mortgage applicant and calling them back, or texting a customer about a confusing bill before it becomes a dispute. The company also introduced ElevenAI, an AI assistant for work that supports dictation into any app, automated meeting notes, context passing between agents and people, and live assistance during calls.
The announcements position ElevenLabs beyond voice synthesis into full conversational agent infrastructure and workplace productivity, putting it in more direct competition with general AI assistant offerings and enterprise agent platforms.
Today at ElevenLabs Summit NYC, we shared what we’re building next for agents. Here’s what we announced: Fine-tuning: In the coming months, enterprises can work with our team to continually fine-tune our Text to Speech and Speech to Text models, as well as open-source LLMs, on their own data. Interaction Engine: A new system that picks the right LLM for each moment of a conversation. It uses less expensive models for simple requests and frontier models for complex ones like refunds, so you get the right outcome without overspending on tokens. Memory in ElevenAgents: Agents remember what gets mentioned in passing, so businesses can act on it. Routines in ElevenAgents: Agents can now run operations before and after conversations, like pre-screening a mortgage applicant and calling them back, or texting about a confusing bill before it turns into a dispute. ElevenAI: A new AI assistant for work that lets you dictate into any app, takes meeting notes, passes context between agents and people, and provides live assistance during calls.
This report is categorized under AITOP's "Products" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include ElevenLabs, ElevenAgents, voice agents, fine-tuning.
Source attribution: curated from ElevenLabs (@elevenlabsio).
