On the same 1,000 Big Bench Hard reasoning questions, GPT-4o answered 92% correctly when they arrived as text, but its real-time speech-to-speech version answered only 66% when the questions were spoken (Artificial Analysis, Big Bench Audio launch 2024). The gap has not closed everywhere: in the 2026 tau-Voice customer-service benchmark, a GPT-5 text agent completed 85% of tasks while full-duplex voice agents completed 31% to 51% with clean audio and 26% to 38% with realistic noise and accents (Ray et al., tau-Voice, arXiv 2026). The weakest skill is not words but voice itself: GPT-4o-Audio got 21.44% of paralinguistic perception questions right in MMSU, against 59.70% for meaning-based questions (Wang et al., MMSU, ICLR 2026). In cascades, the risk starts at the speech recognition front end, where mistakes and invented transcript text flow straight into the answer. We aggregated data from S2SBench, AfriVox, SD-QA, Spoken SQuAD, the OpenAI GPT-4o System Card, the Qwen2-Audio and Kimi-Audio technical reports, and more than a dozen ACL, ICLR and arXiv benchmark papers and leaderboards.
In this roundup, a “speech language model” (also called a speech LLM or large audio language model) is a model that takes audio in directly. A “cascade” is the older design: a speech recognizer writes a transcript, and a text-only LLM answers from that transcript.
TL;DR
- GPT-4o scored 92% on text questions and 66% speech-to-speech on Big Bench Audio (Artificial Analysis, Big Bench Audio launch 2024).
- DiVA dropped from 81.08 with text input to 55.70 with the same instructions spoken (Chen et al., VoiceBench, arXiv v3 2024).
- Baichuan-Audio’s two-stage training cut the spoken story-completion gap to 3.4 points behind a text baseline (S2SBench, arXiv 2025).
- A Whisper plus Llama-3.1-8B cascade scored 55.25% on VoxEval against 37.63% for the best end-to-end model (Cui et al., VoxEval, ACL 2025).
- Gemini 2.0 Flash reached 65.6% on MMAR, beating every caption-plus-LLM cascade, the best of which scored 55.5% (MMAR benchmark paper, arXiv 2025).
- GPT-4o-Audio scored 21.44% on paralinguistic perception vs 59.70% on semantic perception (Wang et al., MMSU, ICLR 2026).
- A Whisper plus GPT-4 cascade identified speaker gender correctly only 21.9% of the time (AIR-Bench, ACL 2024).
- Adding accents cut voice-agent retail task success from 55% to 44% on average (Ray et al., tau-Voice, arXiv 2026).
- EXAONE-3.5-32B fell from F1 0.863 on clean audio to 0.586 at -10 dB signal-to-noise ratio in a Korean cascade (Jung and Choi, arXiv 2026).
- Dynamic-SUPERB grew from 55 evaluation instances to 180 tasks (Dynamic-SUPERB, ICASSP 2024; Dynamic-SUPERB Phase-2, ICLR 2025).
1. How Big Spoken QA Benchmarks Have Become
Benchmark size grew fast between 2023 and 2026, but in two different directions. Dynamic-SUPERB went from 55 evaluation instances (built from 33 tasks and 22 datasets) to 180 tasks in Phase-2, with 125 of them contributed by the research community (Dynamic-SUPERB Phase-2, ICLR 2025). That is breadth: many short tests, each probing one skill. VoxEval took the opposite route and went deep on one skill, knowledge questions, by turning 56 MMLU subjects into 13,938 spoken question-answer pairs, then generating each pair under 11 audio conditions for 153,318 variations (Cui et al., VoxEval, ACL 2025).
The practical takeaway for anyone reading a model card: a single headline score can come from 1,000 items or from 19,000 items, and from synthetic voices or real recordings. Big Bench Audio and MMAR each have 1,000 items, which makes a 1 to 2 point difference between two models hard to read as a real ranking.
| Metric | Value | Source |
|---|---|---|
| Dynamic-SUPERB Phase 1: evaluation instances, tasks and datasets | 55 instances, 33 tasks, 22 datasets | Dynamic-SUPERB, ICASSP 2024 |
| AIR-Bench foundation benchmark: tasks and single-choice questions | 19 tasks, about 19,000 questions | AIR-Bench, ACL 2024 |
| AIR-Bench chat benchmark: open-ended questions, of which about speech | 2,000 questions, 800 about speech | AIR-Bench, ACL 2024 |
| MMAU: audio clips and distinct skills tested | 10,000 clips, 27 skills | MMAU benchmark paper, arXiv 2024 |
| MMSU: questions and tasks (perception and reasoning) | 5,000 questions, 47 tasks (24 perception, 23 reasoning) | Wang et al., MMSU, ICLR 2026 |
| VoiceBench (journal version): spoken instructions and tasks | 6,783 instructions, 8 tasks | Chen et al., VoiceBench, TACL 2026 |
| SD-QA: audio prompts, dialects, speakers and languages | more than 68,000 prompts, 24 dialects, 255 speakers, 5 languages | Faisal et al., SD-QA, EMNLP Findings 2021 |
Context: Big Bench Audio was voiced with 23 synthetic voices taken from text-to-speech models (Artificial Analysis, Big Bench Audio launch 2024), and only around 6.57% of Dynamic-SUPERB Phase-2 audio runs longer than 30 seconds (Dynamic-SUPERB Phase-2, ICLR 2025). Most of these tests therefore measure short questions, not long or messy conversations.
2. Same Questions, Spoken vs Typed
This is the core comparison: identical content, delivered once as text and once as audio, to the same model or model family. DiVA, an end-to-end speech model, scored 81.08 on VoiceBench when given text and 55.70 when the same instructions were spoken, while a Whisper plus LLaMA-3.1-8B cascade lost only 2.37 points (Chen et al., VoiceBench, arXiv v3 2024). The model clearly handles the content; what breaks is the route from sound to meaning.
The newer, more careful designs show the gap can shrink to a few points. S2SBench, from South China University of Technology and Baichuan, measured Baichuan-Audio’s two-stage training at 79.6% on spoken story completion, 3.4 points behind the Qwen2.5-7B text baseline at 83.0% (S2SBench, arXiv 2025). Agent tasks remain far harder: tau-Voice (written τ-Voice in the paper) tests full-duplex voice agents on 278 airline, retail and telecom tasks, and under realistic audio the spoken route still loses more than half of the text capability. Response speed is the other side of that trade-off, covered in our time-to-first-audio statistics.
| Metric | Value | Source |
|---|---|---|
| VoiceBench overall score, Whisper + LLaMA-3.1-8B cascade: text input vs spoken input | 81.43 vs 79.06 | Chen et al., VoiceBench, arXiv v3 2024 |
| VoiceBench overall score, DiVA end-to-end model: text input vs spoken input | 81.08 vs 55.70 | Chen et al., VoiceBench, arXiv v3 2024 |
| Big Bench Audio accuracy: GPT-4o text-to-text vs GPT-4o Realtime Preview speech-to-speech | 92% vs 66% | Artificial Analysis, Big Bench Audio launch 2024 |
| Spoken story completion (sStoryCloze): Qwen2.5-7B with text vs Baichuan-Audio two-stage with speech | 83.0% vs 79.6% | S2SBench, arXiv 2025 |
| Chinese commonsense statements (sCMMLU, 4,743 items): Qwen2.5-7B with text vs GLM-4-Voice instruct model with speech | 70.3% vs 64.3% | S2SBench, arXiv 2025 |
| tau-Voice task success: GPT-5 text agent vs voice agents with clean audio | 85% vs 31% to 51% | Ray et al., tau-Voice, arXiv 2026 |
| tau-Voice: share of GPT-5 text capability kept by voice agents under realistic audio | roughly 30% to 45% | Ray et al., tau-Voice, arXiv 2026 |
Outlier: VoxEval found that one model can score very differently depending on which side is audio. SPIRIT-LM scored 20.84% speech-to-speech, 23.77% speech-to-text and 1.38% text-to-speech on the same questions (Cui et al., VoxEval, ACL 2025), a reminder that the output side matters as well as the input side.
3. Cascade (ASR + LLM) vs End-to-End Speech Models
For questions whose answer lives in the words, the old cascade is still very hard to beat. On VoxEval’s spoken MMLU questions, a Whisper plus Llama-3.1-8B cascade scored 55.25%, while the best end-to-end speech model, GLM-4-Voice, scored 37.63%, and GLM-4-Voice was the only end-to-end model to clear the 25% random-guess line (Cui et al., VoxEval, ACL 2025). Dynamic-SUPERB Phase-2 reports the same pattern: on speech recognition and spoken language understanding, no tested model outperformed the Whisper-LLaMA cascade (Dynamic-SUPERB Phase-2, ICLR 2025).
The picture flips once the answer depends on the sound itself. On MMAR, which mixes speech, music and sound, Gemini 2.0 Flash beat every cascade that first turned audio into a text caption (MMAR benchmark paper, arXiv 2025). The lesson: cascades win when a transcript carries everything; end-to-end models win when the transcript throws away the clue.
| Metric | Value | Source |
|---|---|---|
| VoxEval accuracy: Whisper + Llama-3.1-8B cascade vs GLM-4-Voice end-to-end | 55.25% vs 37.63% | Cui et al., VoxEval, ACL 2025 |
| VoxEval 12-subject subset: gpt-4o-audio-preview vs Whisper + Llama-3.1 cascade | 0.718 vs 0.606 | Cui et al., VoxEval, ACL 2025 |
| AIR-Bench chat, average GPT-4 score (1 to 10): Whisper + GPT-4 cascade vs best end-to-end model (Qwen-Audio Turbo) | 7.54 vs 6.34 | AIR-Bench, ACL 2024 |
| AIR-Bench foundation, average accuracy: Qwen-Audio Turbo end-to-end vs Whisper + GPT-4 cascade | 57.8% vs 53.6% | AIR-Bench, ACL 2024 |
| MMAU test set, speech questions: GPT-4o reading a detailed audio caption vs Gemini Pro v1.5 hearing the audio | 68.66% vs 55.90% | MMAU benchmark paper, arXiv 2024 |
| MMAR average accuracy: Gemini 2.0 Flash end-to-end vs best caption + LLM cascade (caption + DeepSeek-R1) | 65.6% vs 55.5% | MMAR benchmark paper, arXiv 2025 |
| VoiceBench overall score with spoken input: Whisper + GPT-4o cascade vs GPT-4o-Audio | 87.23 vs 86.42 | Chen et al., VoiceBench, arXiv v3 2024 |
Context: AIR-Bench shows both sides in one paper. The cascade won the open-ended chat test but lost the foundation test, and the authors attribute its weak foundation score to questions on speaker gender, age and emotion (see section 4).
4. The Paralinguistic Gap: Hearing How Something Is Said
“Paralinguistic” means everything in a voice beyond the words: emotion, speaker age and gender, speaking style, emphasis. This is where speech LLMs are weakest. In MMSU, GPT-4o-Audio answered 59.70% of semantic perception questions correctly but only 21.44% of paralinguistic perception questions, and Gemini-1.5-Pro, the best model overall, reached only 31.23% on paralinguistic perception (Wang et al., MMSU, ICLR 2026). Perception errors, not reasoning errors, made up 50.3% of GPT-4o-Audio’s mistakes and 51.0% of Gemini-1.5-Pro’s.
Recognizing words has become easy; reading the voice has not. Qwen2-Audio transcribes the LibriSpeech test-clean set at a 1.6% word error rate yet reaches 55.3% accuracy on the MELD emotion recognition test (Qwen2-Audio Technical Report, 2024). For the clinical side of emotion detection, see our speech emotion recognition accuracy statistics.
| Metric | Value | Source |
|---|---|---|
| MMSU perception, GPT-4o-Audio: semantic questions vs paralinguistic questions | 59.70% vs 21.44% | Wang et al., MMSU, ICLR 2026 |
| MMSU perception, Gemini-1.5-Pro: semantic questions vs paralinguistic questions | 57.06% vs 31.23% | Wang et al., MMSU, ICLR 2026 |
| AIR-Bench speaker gender accuracy: Whisper + GPT-4 cascade vs Qwen-Audio Turbo | 21.9% vs 82.5% | AIR-Bench, ACL 2024 |
| AIR-Bench spoken language identification accuracy: Whisper + GPT-4 cascade | 96.8% | AIR-Bench, ACL 2024 |
| Qwen2-Audio: LibriSpeech test-clean word error rate vs MELD emotion accuracy | 1.6% vs 55.3% | Qwen2-Audio Technical Report, 2024 |
| Dynamic-SUPERB Phase-2 emotion recognition, best model (Qwen2-Audio-7B-Instruct) | 75.8% | Dynamic-SUPERB Phase-2, ICLR 2025 |
| Dynamic-SUPERB Phase-2 speaker verification, best system (chance = 50%) | 51.3% | Dynamic-SUPERB Phase-2, ICLR 2025 |
Context: some speaker-identity skills are deliberately turned off. OpenAI reports a score of 0.98 for the deployed GPT-4o on speaker identification requests it should refuse, up from 0.83 for an early version (OpenAI, GPT-4o System Card 2024). A low speaker-identity score can be a safety policy, not only a perception failure.
5. Accents, Noise and Messy Speech
Clean audio flatters every model. In the tau-Voice retail ablation, adding accents alone cut average task success from 55% to 44%, while adding background noise alone cut it to 51% (Ray et al., tau-Voice, arXiv 2026). The accent penalty ranged from 1 point to 18 points by provider, which means a vendor’s clean-audio ranking says little about how it handles your users’ accents.
Noise follows a cliff, not a slope. In a 2026 Korean spoken QA cascade study, answer quality barely moved down to +10 dB signal-to-noise ratio, then fell sharply below 0 dB as the speech recognizer’s character error rate climbed (Jung and Choi, Korean Spoken QA Error Propagation, arXiv 2026). Accent effects on the recognizer itself are covered in our ASR accent bias statistics.
| Metric | Value | Source |
|---|---|---|
| tau-Voice retail task success, average of 3 voice providers: clean audio vs with accents added | 55% vs 44% | Ray et al., tau-Voice, arXiv 2026 |
| tau-Voice retail: drop from adding accents, xAI vs Google | 18 points vs 1 point | Ray et al., tau-Voice, arXiv 2026 |
| Korean spoken QA, EXAONE-3.5-32B F1: typed question vs clean audio vs audio at -10 dB SNR | 0.870 vs 0.863 vs 0.586 | Jung and Choi, arXiv 2026 |
| Korean spoken QA, ASR character error rate: clean vs 0 dB vs -10 dB SNR | 0.0315 vs 0.1143 vs 0.4977 | Jung and Choi, arXiv 2026 |
| VoiceBench: average score loss when instructions contain mispronounced words (all models) | 20.34% | Chen et al., VoiceBench, arXiv v3 2024 |
| VoxEval: Whisper + Llama cascade accuracy change when speaker pitch is altered | minus 9.56 points | Cui et al., VoxEval, ACL 2025 |
| SD-QA English test set, word error rate of a US-English recognizer: US South-East speakers vs Nigerian speakers | 5.25% vs 15.35% | Faisal et al., SD-QA, EMNLP Findings 2021 |
Most recent available data for dialect-level spoken QA: Faisal et al., SD-QA, 2021. On its English test set, passage-selection F1 fell from 58.7 with the original typed question to 55.8 with a US-English ASR transcript. Most recent available data for transcript-only reading comprehension: Spoken SQuAD, 2018, where five reading models averaged F1 74.54 on SQuAD text but 55.40 on Spoken SQuAD ASR transcripts with a 22.73% word error rate (Spoken SQuAD paper, arXiv 2018). Voice diversity testing is also thin: OpenAI compared 3 system voices with 27 English voice samples and called performance on human voices marginally but not significantly worse (OpenAI, GPT-4o System Card 2024). The fix is moving to the model side: fine-tuning Qwen2.5-Omni on three Nigerian languages brought up to a 54% relative reduction in word error rate (Awobade et al., AfriVox, EACL 2026).
6. Human Ceilings and the 2026 Frontier
The hardest audio reasoning benchmarks still leave a wide human margin. Human listeners scored 89.72% on MMSU, while the best of 22 speech and omni models, Gemini-1.5-Pro, scored 60.68% (Wang et al., MMSU, ICLR 2026). On MMAU the margin is similar: 82.23% for humans against 54.90% for Gemini Pro v1.5, the top end-to-end model, on the 1,000-question test-mini split (MMAU benchmark paper, arXiv 2024).
At the same time, the easiest spoken reasoning tests are nearly solved. Big Bench Audio went from 66% for GPT-4o’s real-time model in December 2024 to a 2026 leaderboard where the top entry, StepAudio 3 Realtime, scores 99.7% among 44 models (Artificial Analysis, Speech to Speech leaderboard 2026). Scoring rules changed in between, so the two numbers are not on the same scale, but the direction is clear: at the top of the leaderboard, spoken versions of text-style puzzles are no longer the bottleneck. Voice-specific perception is.
| Metric | Value | Source |
|---|---|---|
| MMAU test-mini: human listeners vs Gemini Pro v1.5 | 82.23% vs 54.90% | MMAU benchmark paper, arXiv 2024 |
| MMAU test set (9,000 questions): Gemini Pro v1.5 vs Qwen2-Audio-Instruct | 52.97% vs 52.50% | MMAU benchmark paper, arXiv 2024 |
| MMAR: Gemini 2.0 Flash vs random guessing | 65.6% vs 29.32% | MMAR benchmark paper, arXiv 2025 |
| MMAR, GPT-4o Audio: speech-only questions vs music-only questions | 70.41% vs 50.97% | MMAR benchmark paper, arXiv 2025 |
| MMAR: models evaluated, open-source plus closed-source | 30 (24 plus 6) | MMAR benchmark paper, arXiv 2025 |
| VoiceBench average reported by the Kimi-Audio team: Kimi-Audio vs GLM-4-Voice | 76.93 vs 57.17 | Kimi Team, Kimi-Audio Technical Report 2025 |
| Kimi-Audio: audio used for pre-training | more than 13 million hours | Kimi Team, Kimi-Audio Technical Report 2025 |
Context: the Kimi-Audio averages are the developer team’s own evaluation, so they are not interchangeable with the VoiceBench authors’ overall scores in sections 2 and 3, because the two papers ran separate evaluations.
Summary: Speech Language Model Spoken Question Answering Benchmarks by the Numbers
| Metric | Value | Source |
|---|---|---|
| Big Bench Audio: GPT-4o text-to-text vs speech-to-speech | 92% vs 66% | Artificial Analysis, Big Bench Audio launch 2024 |
| tau-Voice: GPT-5 text agent vs voice agents with clean audio | 85% vs 31% to 51% | Ray et al., tau-Voice, arXiv 2026 |
| VoiceBench, DiVA end-to-end: text input vs spoken input | 81.08 vs 55.70 | Chen et al., VoiceBench, arXiv v3 2024 |
| S2SBench story completion: text baseline vs best speech model | 83.0% vs 79.6% | S2SBench, arXiv 2025 |
| VoxEval: Whisper + Llama cascade vs GLM-4-Voice | 55.25% vs 37.63% | Cui et al., VoxEval, ACL 2025 |
| MMAR: Gemini 2.0 Flash vs best caption + LLM cascade | 65.6% vs 55.5% | MMAR benchmark paper, arXiv 2025 |
| MMSU, GPT-4o-Audio: semantic vs paralinguistic perception | 59.70% vs 21.44% | Wang et al., MMSU, ICLR 2026 |
| AIR-Bench speaker gender: Whisper + GPT-4 cascade vs Qwen-Audio Turbo | 21.9% vs 82.5% | AIR-Bench, ACL 2024 |
| Qwen2-Audio: LibriSpeech word error rate vs MELD emotion accuracy | 1.6% vs 55.3% | Qwen2-Audio Technical Report, 2024 |
| Korean spoken QA F1 (EXAONE-3.5-32B): clean audio vs -10 dB SNR | 0.863 vs 0.586 | Jung and Choi, arXiv 2026 |
| SD-QA English word error rate: US South-East vs Nigerian speakers | 5.25% vs 15.35% | Faisal et al., SD-QA, EMNLP Findings 2021 |
| Spoken SQuAD reading models, average F1: text vs ASR transcripts | 74.54 vs 55.40 | Spoken SQuAD paper, arXiv 2018 |
| AfriVox: best relative word error rate reduction from fine-tuning Qwen2.5-Omni | up to 54% | Awobade et al., AfriVox, EACL 2026 |
| MMAU test-mini: humans vs Gemini Pro v1.5 | 82.23% vs 54.90% | MMAU benchmark paper, arXiv 2024 |
| Dynamic-SUPERB Phase-2 tasks | 180 (125 new) | Dynamic-SUPERB Phase-2, ICLR 2025 |
| Kimi-Audio pre-training audio | more than 13 million hours | Kimi Team, Kimi-Audio Technical Report 2025 |
| Big Bench Audio 2026 leader score | 99.7% (StepAudio 3 Realtime) | Artificial Analysis, Speech to Speech leaderboard 2026 |
Methodology and Sources
- Chen et al., VoiceBench: Benchmarking LLM-Based Voice Assistants, arXiv v3, December 2024 and its journal version in Transactions of the ACL, volume 14, 2026.
- AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension, ACL 2024.
- Dynamic-SUPERB, ICASSP 2024 and Dynamic-SUPERB Phase-2, ICLR 2025.
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark, arXiv 2024.
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix, arXiv 2025.
- Wang et al., MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark, ICLR 2026.
- Cui et al., VoxEval, ACL 2025.
- S2SBench: Quantifying Intelligence Degradation in Speech-to-Speech LLMs, South China University of Technology and Baichuan, arXiv 2025.
- Ray et al., tau-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains, Sierra and Princeton University, arXiv March 2026.
- Jung and Choi, Analyzing Error Propagation in Korean Spoken QA with ASR-LLM Cascades, arXiv 2026.
- Faisal et al., SD-QA: Spoken Dialectal Question Answering for the Real World, Findings of EMNLP 2021.
- Spoken SQuAD: Mitigating the Impact of Speech Recognition Errors on Listening Comprehension, arXiv 2018.
- Awobade et al., AfriVox: Probing Multilingual and Accent Robustness of Speech LLMs, EACL 2026.
- Artificial Analysis, Evaluating Audio Reasoning with Big Bench Audio, December 20, 2024, Speech to Speech leaderboard 2026 and speech-to-speech benchmarking methodology.
- OpenAI, GPT-4o System Card, 2024.
- Qwen2-Audio Technical Report, 2024 and Kimi Team, Kimi-Audio Technical Report, 2025.
- Data watch: Big Bench Audio scoring changed from v1.0 (unanswered questions not penalized, December 2024 to March 2026) to v1.1 (all 1,000 questions counted, March 2026) and v1.2 (updated judge and grader, May 2026), so the 66% (2024) and 99.7% (2026) figures are not on the same scale. The leaderboard page gives no date for its scores, and its chart plots 37 of the 44 models; we read it on October 31, 2026. VoiceBench scores here come from the December 2024 preprint, whose text and tables disagree in places (the text says the cascade lost 4.37 points, the table shows 2.37; we used the table), and the 2026 journal version may report different scores. VoxEval’s text reports drops of up to 6% from speaking style and audio quality, while its table shows minus 9.56 points for pitch; we used the table. The MMSU paper’s prose and Table 4 disagree on which GPT-4o-Audio error category is 14.7%. MMAR says 30 models, while its table lists 31 rows. Dynamic-SUPERB Phase-2 states no model beats chance on query-by-example search, while its table shows 53.5% for one model. The S2SBench figures for GLM-4-Voice on sCMMLU use the instruct model because no base model was provided. SD-QA (2021) and Spoken SQuAD (2018) are older than three years and cover cascade systems only. tau-Voice tested 3 commercial voice agents, so its spread may change quickly as providers update models, and the MMSU, MMAU and Kimi-Audio tables mostly cover 2024 and 2025 model generations.
Last updated: October 31, 2026. We update this roundup quarterly, and the next refresh is expected when Artificial Analysis publishes its next Speech to Speech Index version and the ICLR 2027 speech LLM benchmark papers appear.