At its New York Summit on November 11, ElevenLabs laid out its next phase for agents. Available now: NVIDIA Nemotron 3.5 Super VL in the ElevenAgents model list, the ElevenLabs Partner Program with a McKinsey alliance, and free early access to a new app called ElevenAI. Previewed or coming in the next few months: Interaction Engine, which manages turn-taking and interruptions and picks the right LLM for each step (cheap models for triage, frontier models for complex steps such as refunds); Memory and Routines in ElevenAgents; and fine-tuning of open-source LLMs and ElevenLabs voice models on enterprise data.
Key Takeaways
- ✓Live now: NVIDIA Nemotron 3.5 Super VL in ElevenAgents; on the ElevenAgents customer-support benchmark it leads end-to-end task completion at 70% lower cost than other leading open models, and with reasoning off it ranks among the 3 fastest models on the platform (vendor figures)
- ✓Preview: Interaction Engine handles turn-taking and interruptions and switches between cheap and frontier LLMs per conversation step to save tokens
- ✓Coming: Memory keeps preferences and constraints returning customers mentioned; Routines run multi-step work before, during and after calls (pre-screening, callbacks, CRM writes, email reports)
- ✓Ecosystem: 550+ partners in 44 countries; from next year the Marketplace lets buyers pay for industry agents with existing ElevenLabs commitments
- ✓Compute: 15 MW of new NVIDIA Vera Rubin capacity; an on-prem voice agent for Poland's national e-Health platform (about 40 million appointments a year)
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background and pain points
Voice and chat agents are moving from demos to production, and enterprises now care about per-call token cost, remembering returning customers and who does the follow-up work. ElevenLabs framed its summit around turning the conversation layer into enterprise infrastructure; CEO Mati Staniszewski said conversational agents will become as essential as cloud or payments.
Architecture and mechanisms
Interaction Engine starts with conversational rhythm, handling turn-taking and interruptions, and picks an LLM per step: cheap models for triage, frontier models for complex steps. Memory distills each call into a structured record that captures details a CRM has no field for. Routines is a new automation layer that connects to existing systems and runs multi-step work around calls. ElevenLabs also plans to fine-tune some open-source LLMs and its own TTS/STT models with customers using live deployment data, following the Scribe v3 Medical approach.
Benchmarks
The only number is for Nemotron: on the ElevenAgents customer-support benchmark, Nemotron 3.5 Super VL leads end-to-end completion at 70% lower cost than other leading open models (vendor figures; rivals and scores not disclosed). No data has been published for Interaction Engine, Memory or Routines.
Getting started
Today you can switch an ElevenAgents agent to Nemotron 3.5 Super VL and compare cost and completion on your own call samples, and request free early access to ElevenAI. Interaction Engine, Memory, Routines and fine-tuning are previews or coming in the next few months with no dates or pricing yet.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.