The winning system in the most recent DCASE audio captioning task scored FENSE 0.554, only 0.044 above the organizers’ baseline of 0.510, and 29 of the 30 team systems beat that baseline (DCASE Challenge, Task 6 Results 2024). Humans still sit far ahead on the harder dataset: reference captions agree with each other at SPIDEr 0.54 on Clotho (Mei et al., Audio Captioning Overview, 2022), while a compact model reaches 0.301 (Labbé et al., CoNeTTE, 2023). The measuring stick is imperfect too, because the FENSE metric picks the caption humans prefer in 75.7% of Clotho-Eval pairs (Zhou et al., FENSE, 2021). This roundup covers machines describing non-speech sounds, not speech subtitles, which are covered in our captions and subtitles statistics and our captioning and subtitling market roundup. We aggregated data from the Clotho, AudioCaps, WavCaps and AudioSetCaps dataset papers, the FENSE, MACE, CLAIR-A and BRACE evaluation studies, the Audio Flamingo and MMAU-Pro papers, OpenAlex, and the challenge’s own results tables.
A few terms recur below. A “caption” here is a machine-written or human-written sentence describing sounds. A “baseline” is the reference system supplied by the organizers, which teams try to beat. SPIDEr, CIDEr and SPICE are scores borrowed from image captioning, FENSE combines sentence similarity with a penalty for broken grammar, and SPIDEr-FL is SPIDEr with a similar penalty. For all of them, higher is better.
TL;DR
- The 2024 winner beat the baseline on FENSE, 0.554 vs 0.510, but trailed it on SPIDEr-FL, 0.241 vs 0.287 (DCASE Challenge, Task 6 Results 2024).
- Human reference captions agree at SPIDEr 0.56 on AudioCaps and 0.54 on Clotho (Mei et al., Audio Captioning Overview, 2022).
- One model scored SPIDEr 0.466 on AudioCaps but 0.301 on Clotho (Labbé et al., CoNeTTE, 2023).
- FENSE matched human preference in 75.7% of Clotho-Eval pairs; SPICE matched 56.3% (Zhou et al., FENSE, 2021).
- On pairs of two correct human captions, SPICE scored 44.3% on Clotho-Eval, worse than a coin flip (Dixit et al., MACE, 2024).
- WavCaps holds 403,050 clips and 7,567.92 hours of audio (Mei et al., WavCaps, 2023).
- AudioSetCaps holds 1,910,920 captions, rated 3.70 out of 5 by humans (Bai et al., AudioSetCaps, 2024).
- GPT-4o-Audio-Preview scored F1 95.76 and 96.75 at spotting invented sounds in captions, but 55.24 at ranking caption quality (Guo et al., BRACE, 2025).
- Papers with ‘audio captioning’ in the title rose from 6 in 2019 to 36 in 2024 (OpenAlex, title search, October 31, 2026).
1. Leaderboard Scores: Five Editions, Shrinking Margins
The story of this leaderboard is a rising floor. In SPIDEr terms, the winner’s margin over the baseline went from 0.298 in 2021 (0.310 minus 0.012) to 0.095 in 2022 (0.319 minus 0.224), because the organizers’ starting system improved far faster than the top of the table. By 2024 the baseline ranked 30th of 31 entries, and the whole field fit between FENSE 0.508 and 0.554.
The ranking is also close to a statistical tie at the top. The winner’s 95% interval (0.543 to 0.563) overlaps the runner-up’s (0.534 to 0.555), although it clears the baseline’s (0.499 to 0.521). Change the metric and the order changes: the winning entry sampled candidate captions, filtered them and summarized them with GPT-4 Turbo, which suited FENSE far better than SPIDEr-FL. Section 5 covers how far each metric can be trusted.
| Metric | Value | Source |
|---|---|---|
| Best vs baseline SPIDEr on the held-out ranking split, by edition | 2020: 0.222 vs 0.018; 2021: 0.310 vs 0.012; 2022: 0.319 vs 0.224 | DCASE 2020, 2021 and 2022 Task 6 results |
| 2023 best vs baseline SPIDEr-FL on the held-out ranking split | 0.327 vs 0.264 | DCASE 2023 Task 6a results |
| 2024 best vs baseline FENSE on the Clotho evaluation split | 0.554 (95% CI 0.543 to 0.563) vs 0.510 (95% CI 0.499 to 0.521) | DCASE 2024 Task 6 results |
| 2024 winner vs baseline SPIDEr-FL; highest SPIDEr-FL of any 2024 entry | 0.241 vs 0.287; highest 0.329, from a system flagged for a data leak | DCASE 2024 Task 6 results |
| One team’s single model vs its 25-model ensemble, FENSE in its own report | 0.5410 vs 0.5442 (baseline 0.5040) | Jung et al., technical report 2024 |
Most recent available data: DCASE Challenge, Task 6 Results 2024. No later edition has a captioning task. The ensemble row shows how little brute force was left to gain: 25 models added 0.0032 FENSE (0.5442 minus 0.5410).
2. Participation and Publication Volume
Team counts stayed flat: the task held between 9 and 12 teams for five years, then left the program. Neither the 2025 program (six tasks) nor the 2026 program (seven tasks) lists audio captioning. The published pages do not say why, but the replacement tasks hint at the direction, and section 7 follows it.
Academic output tells a similar story. Papers with the phrase in their title multiplied by six between 2019 and 2024 (36 divided by 6), then flattened. A broader OpenAlex search keeps climbing, which suggests that captioning is becoming one feature inside larger audio models rather than a field of its own. That reading is ours, and the broad search is noisy.
| Metric | Value | Source |
|---|---|---|
| Teams and submitted systems per edition, excluding the baseline | 2020: 10 and 34; 2021: 12 and 37; 2022: 9 and 36; 2023: 10 and 28; 2024: 11 and 30 | DCASE Task 6 results 2020 to 2024 (links in section 1) |
| Captioning tasks in the challenge program; submission entries across the 2025 program | 2025: 0 of 6 tasks; 2026: 0 of 7 tasks; 277 entries in 2025 | DCASE 2025 and DCASE 2026 program pages |
| Papers with ‘audio captioning’ in the title, by publication year | 2019: 6; 2022: 27; 2024: 36; 2025: 33; 2026 to October 31: 29 | OpenAlex title search, October 31, 2026 |
| All papers with ‘audio captioning’ in the title, 2017 to October 31, 2026 | 203 | OpenAlex title search, October 31, 2026 |
| Records returned by the broader OpenAlex phrase search | 2,109 in total; 433 in 2025; 726 in 2026 to October 31 | OpenAlex general search, October 31, 2026 |
3. The Human Ceiling: How Far Machines Sit From Human Agreement
Audio captioning has no single right answer, so researchers score the human references against each other to estimate a ceiling, called a consensus score. Humans agree at nearly the same level on both main datasets, SPIDEr 0.56 on AudioCaps and 0.54 on Clotho, yet the same model architecture lands far apart: 0.466 and 0.301.
Put as a share of the human figure, that is about 83% on AudioCaps (0.466 divided by 0.56) and 56% on Clotho (0.301 divided by 0.54). The gap is therefore not explained by human disagreement alone. Clotho clips are also longer, 15 to 30 seconds against 10, and MACS shows the other extreme, where human captions of the same clip barely overlap in wording. Its Sentence-BERT figure holds up far better than its SPIDEr figure, which suggests people describe the same scene in different words.
The last row matters for anyone reading AudioCaps claims. Encoders pretrained on AudioSet have already heard the AudioCaps test files, and removing those files from pretraining cost one model 0.029 SPIDEr.
| Metric | Value | Source |
|---|---|---|
| Human consensus SPIDEr on AudioCaps, Clotho and MACS | 0.56, 0.54 and 0.17 | Mei et al., Audio Captioning Overview, 2022 |
| Human consensus CIDEr on AudioCaps, Clotho and MACS | 0.90, 0.86 and 0.21 | Mei et al., Audio Captioning Overview, 2022 |
| Human consensus Sentence-BERT on AudioCaps, Clotho and MACS | 0.64, 0.53 and 0.52 | Mei et al., Audio Captioning Overview, 2022 |
| Same CNext-trans architecture, SPIDEr on the AudioCaps test set vs the Clotho test set | 0.466 vs 0.301 | Labbé et al., CoNeTTE, 2023 |
| Same CNext-trans architecture, FENSE on the AudioCaps test set vs the Clotho test set | 0.633 vs 0.516 | Labbé et al., CoNeTTE, 2023 |
| CNext-trans SPIDEr on AudioCaps, encoder that had heard the test audio vs encoder that had not | 0.495 vs 0.466 | Labbé et al., CoNeTTE, 2023 |
Most recent available data: Mei et al., Audio Captioning Overview, 2022. The consensus figures use different subsets from the model scores: AudioCaps validation and test, Clotho training, and MACS clips with five references.
4. Dataset Scale: From Crowdsourced to LLM-Written Captions
Human annotation built the field, and large language models (LLMs) scaled it. WavCaps holds 403,050 clips and about 60 times the audio of the AudioCaps training set (7,563.3 hours divided by 126.6 hours, both from Labbé et al., 2023). AudioSetCaps then pushed the count to 1,910,920 captions, with mean caption length rising from 11 words in Clotho to 28.
More data did not mean free gains. WavCaps kept only 46.3% of its FreeSound clips (262,300 of 567,078) after cleaning, and pretraining on it lifted scores by a modest amount. Raw quality also held up better than the “weakly labelled” name suggests, although 79 clips is a small sample.
| Metric | Value | Source |
|---|---|---|
| Clotho version 1 size | 4,981 clips, 24,905 captions, 15 to 30 seconds per clip | Drossos et al., Clotho, 2019 |
| AudioCaps official size | 51,308 clips of 10 seconds | Mei et al., Audio Captioning Overview, 2022 |
| AudioCaps training files still available to one research team | 46,213 of 49,838 (92.7%) | Labbé et al., CoNeTTE, 2023 |
| WavCaps size | 403,050 clips, 7,567.92 hours | Mei et al., WavCaps, 2023 |
| WavCaps FreeSound clips before vs after cleaning with ChatGPT | 567,078 vs 262,300 | Mei et al., WavCaps, 2023 |
| SPIDEr (percent) when trained only on the target dataset vs pretrained on WavCaps and fine-tuned on the target | HTSAT-BART on AudioCaps: 44.4 vs 48.5; CNN14-BART on Clotho: 25.5 vs 31.0 | Mei et al., WavCaps, 2023 |
| AudioSetCaps caption entries; captions across its three extended datasets | 1,910,920; 6,117,099 | Bai et al., AudioSetCaps, 2024 |
| Human mean opinion score out of 5 across 79 clips | AudioSet labels 3.81; AudioCaps 3.79; AudioSetCaps 3.70; WavCaps 3.70; Auto-ACD 3.60 | Bai et al., AudioSetCaps, 2024 |
Most recent available data for Clotho: Drossos et al., 2019. AudioCaps clips are YouTube-derived, so availability shrinks as videos disappear, and two teams may not be training on the same files.
5. Evaluation Metrics: Do the Scores Match Human Judgment?
A leaderboard is only as good as its metric, and the metrics used for the first years of this task were never built for sound. FENSE, the 2024 ranking metric, still disagrees with human preference in about one Clotho-Eval pair in four (100 minus 75.7 gives 24.3%). SPICE fares worse, and on pairs of two correct human captions it falls below chance.
The human side is shaky too. When raters scored captions on an absolute scale, agreement was poor, with Fleiss kappa (a measure of rater agreement, where 1 is perfect) of 0.18 on Clotho. Asking which of two captions is better lifted it to 0.48. That is why these benchmarks compare pairs. The 2024 generation of metrics adds audio or an LLM judge and gains a few more points, and CLAIR-A also explains its scores in words, which human raters found fairer than a bare FENSE number.
| Metric | Value | Source |
|---|---|---|
| Share of pairs where the metric picked the human-preferred caption, FENSE vs SPICE | Clotho-Eval: 75.7% vs 56.3%; AudioCaps-Eval: 85.3% vs 59.7% | Zhou et al., FENSE, 2021 |
| Rater agreement (Fleiss kappa), absolute scores vs pairwise choices | Clotho: 0.18 vs 0.48; AudioCaps: 0.23 vs 0.33 | Zhou et al., FENSE, 2021 |
| Size of the Clotho-Eval and AudioCaps-Eval benchmarks | 1,750 and 1,671 pairs; 250 clips each; 4 raters per pair | Zhou et al., FENSE, 2021 |
| FENSE error detector at threshold 0.9, on 723 machine captions | Precision 96.8, recall 76.4, F1 85.4 | Zhou et al., FENSE, 2021 |
| Accuracy on Clotho-Eval pairs of two correct human captions: SPICE, FENSE, MACE | 44.3%, 60.5%, 63.3% | Dixit et al., MACE, 2024 |
| Overall Clotho-Eval accuracy: CLAIR-A with GPT-4o vs FENSE | 79.7% vs 75.7% | Wu et al., CLAIR-A, 2024 |
| Human fairness rating out of 5 for FENSE scores vs CLAIR-A scores with written reasons (N=200) | 2.97 vs 3.66 | Wu et al., CLAIR-A, 2024 |
Most recent available data for the pair benchmarks themselves: Zhou et al., 2021. Newer studies rerun new metrics on these same 2021 pairs.
6. Large Audio Models, Model Size and Invented Sounds
Captioning has moved from a dedicated task to a side skill of general audio models, and the specialists still lead on their home ground. Audio Flamingo 3 lifted CIDEr on AudioCaps from 0.58 to 0.70, yet that remains well under the 0.90 human consensus from section 3 and the 84.8 (on a 0 to 100 scale) reported for a model trained on AudioSetCaps and tuned on AudioCaps. Size is no guarantee either: a 12.0M-parameter model competes with systems of 171M and 400M.
Good captions do not substitute for hearing. When MMAU-Pro fed captions to GPT-4o instead of audio, accuracy fell to 35.3%, against 59.2% for a model that takes the audio directly. Readers tracking the same split in speech can compare our spoken question answering benchmarks.
The BRACE results show a split personality. GPT-4o-Audio-Preview was the best at catching a caption with one invented noun but weak at ranking two real captions. The risk mirrors speech recognition hallucination, with invented sounds in place of invented words.
| Metric | Value | Source |
|---|---|---|
| CIDEr, Audio Flamingo 2 vs Audio Flamingo 3 | AudioCaps: 0.58 vs 0.70; Clotho-v2: 0.46 vs 0.50 | Goel et al., Audio Flamingo 3, 2025 |
| Model pretrained on AudioSetCaps, AudioCaps and Clotho, then fine-tuned on AudioCaps: scores on the AudioCaps test set | CIDEr 84.8, SPIDEr 51.6 | Bai et al., AudioSetCaps, 2024 |
| Trainable parameters on AudioCaps: CNext-trans vs HTSAT-BART vs PYB | 12.0M vs 171M vs 400M | Labbé et al., CoNeTTE, 2023 |
| Ranking caption quality on 2,496 pairs (BRACE-Main), F1: best CLAP-based metric vs Audio Flamingo 2 vs GPT-4o-Audio-Preview | 70.01 vs 63.19 vs 55.24 | Guo et al., BRACE, 2025 |
| Spotting a caption with one swapped noun on 10,315 pairs (BRACE-Hallucination), F1 on AudioCaps and Clotho: GPT-4o-Audio-Preview vs Audio Flamingo 2 | 95.76 and 96.75 vs 79.55 and 72.91 | Guo et al., BRACE, 2025 |
| MMAU-Pro average on 5,305 questions: captions plus GPT-4o vs Audio Flamingo 3 vs Gemini 2.5 Flash | 35.3 vs 51.7 vs 59.2 | Kumar et al., MMAU-Pro, 2025 |
| Audio question answering accuracy, best entry vs strongest baseline (two different benchmarks) | 2025: 73.74% vs 52.09%; 2026: 58.33% vs 53.17% | DCASE 2025 Task 5 and DCASE 2026 Task 5 results |
7. Timestamped Captions: Where the Benchmark Moved
A sentence for a whole clip cannot say when something happened, and the newest datasets exist to fix that. The 2026 task on finding a moment in long audio drew 21 teams and 59 submissions, more than any of the five captioning editions in section 2, and the top Recall1@0.7 of 48.59 was over three times the 13.56 baseline (48.59 divided by 13.56 gives 3.6). Recall1@0.7 asks whether the model’s top guess overlaps the true moment by at least 0.7.
The gap between the task’s top result and the 20.3 reported in the dataset paper is large, but the two were measured on different test sets. TACOS adds a second signal: when training moved from clip-level to time-aligned captions, text-based sound detection scored higher, although it stayed far under supervised training.
| Metric | Value | Source |
|---|---|---|
| TACOS size | 12,358 recordings; 47,748 time-aligned captions; 76.59 hours | Primus et al., TACOS, 2025 |
| TACOS annotation | 337 students; 3.57 captions per clip on average; 92.84% of each clip covered on average | Primus et al., TACOS, 2025 |
| Text-based sound detection on 50 categories, PSDS1 (a detection score): pretraining, clip-level fine-tuning, time-aligned fine-tuning, fully supervised | 4.61, 12.44, 17.99, 54.26 | Primus et al., TACOS, 2025 |
| CASTELLA size | 1,862 recordings of 1 to 5 minutes; 3,881 local captions; 11,308 timestamps; over 120 hours | Munakata et al., CASTELLA, 2025 |
| Clotho-Moment size, the synthetic companion dataset | 44.3K annotated clips; 853 hours | Munakata et al., CASTELLA, 2025 |
| Best Recall1@0.7 in the CASTELLA paper | 20.3 | Munakata et al., CASTELLA, 2025 |
| 2026 Audio Moment Retrieval task: teams, submissions, best vs baseline Recall1@0.7 | 21 teams; 59 submissions; 48.59 vs 13.56 | DCASE 2026 Task 6 results |
Summary: Automated Audio Captioning by the Numbers
| Metric | Value | Source |
|---|---|---|
| 2024 winner vs baseline | FENSE 0.554 vs 0.510; SPIDEr-FL 0.241 vs 0.287 | DCASE Challenge, Task 6 Results 2024 |
| Human consensus SPIDEr on AudioCaps and Clotho | 0.56 and 0.54 | Mei et al., Audio Captioning Overview, 2022 |
| CNext-trans SPIDEr, AudioCaps test set vs Clotho test set | 0.466 vs 0.301 | Labbé et al., CoNeTTE, 2023 |
| Clotho version 1 | 4,981 clips, 24,905 captions | Drossos et al., Clotho, 2019 |
| AudioCaps official size | 51,308 clips | Mei et al., Audio Captioning Overview, 2022 |
| WavCaps size | 403,050 clips, 7,567.92 hours | Mei et al., WavCaps, 2023 |
| AudioSetCaps caption entries | 1,910,920 | Bai et al., AudioSetCaps, 2024 |
| Human mean opinion score out of 5, AudioCaps vs AudioSetCaps | 3.79 vs 3.70 | Bai et al., AudioSetCaps, 2024 |
| Clotho-Eval accuracy, FENSE vs SPICE | 75.7% vs 56.3% | Zhou et al., FENSE, 2021 |
| Clotho rater agreement (Fleiss kappa), absolute scores vs pairwise choices | 0.18 vs 0.48 | Zhou et al., FENSE, 2021 |
| Accuracy on pairs of two correct human captions, SPICE on Clotho-Eval | 44.3% | Dixit et al., MACE, 2024 |
| Overall Clotho-Eval accuracy, CLAIR-A with GPT-4o | 79.7% | Wu et al., CLAIR-A, 2024 |
| CIDEr on AudioCaps, Audio Flamingo 2 vs Audio Flamingo 3 | 0.58 vs 0.70 | Goel et al., Audio Flamingo 3, 2025 |
| GPT-4o-Audio-Preview F1, ranking caption quality vs spotting a swapped noun on AudioCaps | 55.24 vs 95.76 | Guo et al., BRACE, 2025 |
| MMAU-Pro average, captions plus GPT-4o vs Gemini 2.5 Flash | 35.3 vs 59.2 | Kumar et al., MMAU-Pro, 2025 |
| TACOS time-aligned captions | 47,748 | Primus et al., TACOS, 2025 |
| CASTELLA local captions | 3,881 | Munakata et al., CASTELLA, 2025 |
| Papers with ‘audio captioning’ in the title, 2024 vs 2025 | 36 vs 33 | OpenAlex title search, October 31, 2026 |
Methodology and Sources
We read the results tables, task pages and papers below during this update and cite the original authors. No figure comes from a blog or a leaderboard mirror.
- DCASE Challenge results: 2020, 2021, 2022, 2023, 2024, 2025 Task 5, 2026 Task 5, 2026 Task 6; task pages: 2020, 2022, 2024, 2026 Task 6; programs: 2025, 2026.
- Jung et al., encoder fusion and LLM summarization technical report, 2024.
- Drossos et al., Clotho: An Audio Captioning Dataset, 2019.
- Mei et al., Automated Audio Captioning: An Overview of Recent Progress and New Challenges, 2022.
- Mei et al., WavCaps, 2023.
- Bai et al., AudioSetCaps, 2024.
- Labbé et al., CoNeTTE, 2023.
- Zhou et al., Can Audio Captions Be Evaluated with Image Caption Metrics?, 2021.
- Dixit et al., MACE, 2024.
- Wu et al., CLAIR-A, 2024.
- Guo et al., BRACE, 2025.
- Goel et al., Audio Flamingo 3, 2025.
- Kumar et al., MMAU-Pro, 2025.
- Primus et al., TACOS, 2025.
- Munakata et al., CASTELLA, 2025.
- OpenAlex title search and general search, both run on October 31, 2026.
- Data watch: scores across editions are not strictly comparable. The ranking metric changed from SPIDEr (2020 to 2022) to SPIDEr-FL (2023) to FENSE (2024), the 2020 edition used Clotho version 1 while the 2022 and 2024 editions used version 2, and the held-out split was called the testing split until 2023 and the evaluation split in 2024. Clotho totals differ by edition: the 2022 task page gives 6,974 clips and 34,870 captions, the 2024 task page 6,972 and 34,860. CoNeTTE’s text gives 3,840 Clotho training files and its table 3,839, and its conclusion cites a 49.5% figure that matches the overlapping encoder in its Table V, so we quote the table. WavCaps is 7,567.92 hours in its own paper and 7,563.3 in CoNeTTE. MACE’s prose appears to swap its Clotho-Eval and AudioCaps-Eval figures for FENSE, so we follow the original FENSE paper; CLAIR-A’s abstract claims a 5.8% gain over FENSE while its table shows 4.0 points, so we quote the table. BRACE’s Avg-All for GPT-4o-Audio-Preview on hallucination (96.37) does not equal the mean of its two columns, so we quote the columns; its swapped-noun captions were written by GPT-4o, the same model family as the best detector, and its filtered pairs keep only unanimous human votes. AudioSetCaps gives ‘at least five’ raters in its text and a different figure in a figure caption; we did not confirm whether AudioCaps test clips were excluded from its AudioSet-based training. CIDEr is reported on 0 to 1 and 0 to 100 scales, and Audio Flamingo 2 labels its AudioCaps score zero-shot, so cross-paper CIDEr comparisons are approximate. Clotho-Moment is listed as 44.3K clips in the CASTELLA paper and as over 51,240 recordings with 44,261 captions on the 2026 task page. OpenAlex counts change as records are added, and 2026 is a partial year. Most recent available data for Sections 3 to 5 includes 2019 to 2023 papers; no newer consensus study turned up.
Last updated: October 31, 2026. We update this roundup quarterly, and the next refresh is expected when the DCASE 2027 Challenge publishes its task descriptions.