Hugging Face benchmark tests voice agents on code-switched customer speech
ServiceNow-AI published a Hugging Face benchmark and dataset for code-switched ASR, testing how voice-agent transcription handles Spanish-English, French-English, Canadian French-English, and German-English support scenarios.
ServiceNow-AI published a Hugging Face benchmark for code-switched automatic speech recognition in enterprise voice-agent settings. The benchmark covers Spanish-English, French-English, Canadian French-English, and German-English support scenarios and pairs the blog analysis with a public 918-row dataset. The reported takeaway is narrow but useful: model choice changes downstream reliability when bilingual speakers switch languages inside the same request.
Key takeaways
- The benchmark focuses on ASR, the transcription layer that voice agents depend on before routing, answering, or escalating a request.
- The public dataset has 918 rows across four configurations: Spanish-English, French-English, Canadian French-English, and German-English.
- ServiceNow-AI reports Word Error Rate, Semantic Word Error Rate, and Answer Error Rate to separate raw transcription errors from downstream task failures.
- The blog names ElevenLabs Scribe V2, Google Gemini 3 Flash, and AssemblyAI Universal-3 Pro as top performers across the tested setup.
- The authors warn that the audio is synthetic and that results should be checked against the language pairs and deployment settings a team actually uses.
Practical LinkLoot angle
This is a useful benchmark pattern for anyone building support bots, helpdesk voice flows, or internal IT agents in multilingual teams. Do not evaluate a voice stack only on clean monolingual demos. Test the exact phrases your customers use: product names, ticket IDs, HR terms, acronyms, English technical words inside another language, and mixed-language follow-up questions.
| Evaluation layer | Best use | Limitation | Source |
|---|---|---|---|
| WER | Measures raw transcript distance from the reference | Can overstate harmless spelling differences | Hugging Face Blog |
| SWER | Tracks meaning-changing transcript errors | Depends on a judge model and benchmark design | Hugging Face Blog |
| AER | Tests whether downstream questions can still be answered from the transcript | Needs scenario-specific questions | Hugging Face Blog |
| Public dataset | Lets teams inspect examples and row counts | Synthetic audio, not a full production substitute | Hugging Face Dataset |
For a practical rollout, build a small internal test set before buying a voice-agent platform. Include at least 100 real or realistic turns per major language pair, score both transcript quality and task completion, and keep the failed examples as regression tests when you change ASR providers.
What to verify before you act
Check whether your target ASR provider supports language hints, forced language settings, custom vocabulary, or per-call metadata. ServiceNow-AI evaluated auto language detection, which matches many production calls, but some systems improve when they know the likely languages in advance. Also inspect privacy terms before uploading support audio, because bilingual helpdesk recordings can include employee IDs, contact details, health leave terms, or customer account information.
The benchmark is synthetic, so do not treat the ranking as a universal production answer. Use it to shortlist providers and metrics, then run a smaller test on your own accents, call quality, microphones, background noise, and domain vocabulary.
Source check
The Hugging Face blog confirms the benchmark design, language pairs, evaluation metrics, named model results, and limitations. The Hugging Face dataset page corroborates the public dataset and row counts. daily.dev independently surfaced the story as a developer-news item, but the technical claims here are taken from the primary blog and dataset.
It is speech where a user switches languages inside the same request, such as German support wording mixed with English IT terms.
For more workflow design ideas, connect this benchmark with LinkLoot's guide to AI workflow automation before choosing a voice-agent stack.