🌐 Website • 🤗 Hugging Face • 📃 arXiv Paper

• August 21, 2025: C3 Benchmark has been accepted to EMNLP 2025 main conference.

• 🌍 Bilingual Coverage: Comprehensive evaluation in both English and Chinese.

• 🎯 Real-world Complexity: Based on empirical analysis of actual spoken dialogues, covering 1,079 instances with 1,586 audio-text paired samples.

• 💪 LLM-based Automatic Evaluation: Reliable evaluation with >0.87 correlation to human judgments using GPT-4o and DeepSeek-R1. See prompts.md for evaluation prompts.

• 🎵 End-to-End Focus: Specifically designed for end-to-end spoken dialogue models, considering crucial phonological features.

• 📊 Challenging Benchmark(as of 29 July 2025): Comprehensive evaluation of 10 leading SDMs reveals the benchmark’s difficulty. Top scores reach only 40.08 % (Chinese) and 55.68 % (English).

1. Prepare Data:

   • Download the C3 dataset to reference_path
   • Organize your SDM responses in answer_path with the structure as the ResponseStructure

2. Run Evaluation:

   • Use evaluate.py for automatic evaluation (see EvaluationUsage.md)

3. Calculate Accuracy:

   • Calculate: Use process_results.py to generate final accuracy metrics automatically, see CalculationUsage.md for detailed workflow

4. Submit Results:

   • Send JSON files from step 3 to chengqianma@yeah.net for Leaderboard inclusion with subject [C3Bench Submission] - [Model_Name]

If you find C3Bench useful for your research and applications, feel free to give us a star ⭐ or cite us using:

@inproceedings{ma2025c3,
 title={C3: A Bilingual Benchmark for Spoken Dialogue Models Exploring Challenges in Complex Conversations},
 author={Ma, Chengqian and Tao, Wei and Guo, Yiwen},
 booktitle={Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
 year={2025},
 publisher={Association for Computational Linguistics},
 doi={10.18653/v1/2025.emnlp-main.1160},
 pages={22789--22807},
 ISBN={979-8-89176-332-6},
}