CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents
Story 6, from arXiv
CR-Bench-v2 is a new benchmark for evaluating AI code review tools, showing that even the best systems detect only about one-third of known defects.
Key points
- CR-Bench-v2 is a defect-centric evaluation framework for AI code review agents.
- It includes 584 instances, with 174 verified defects from SWE-Bench.
- Two metrics are introduced: Known-Defect Detection Rate (KDDR) and Usefulness Rate.
- Top systems detect about 33% of known defects, while Gemini Code Assist detects 8.1%.
- The benchmark uses full repository context and ground-truth fixes for evaluation.
Why it matters
As AI code review tools become more common, understanding their real-world effectiveness is critical. This benchmark highlights the gap between current capabilities and the complexity of real code review tasks.
Counterpoint
The benchmark focuses on defect detection, but real-world code reviews often involve more than just finding bugs—such as assessing design and maintainability, which are harder to quantify.
Sources
This summary was written by a language model and may contain errors. Read the original for the full story.