| Title |
Scorecard of AI benchmark quality |
| Authors |
San Joaquin, Ayrton ; Gipiškis, Rokas ; Chin, Ze Shen |
| DOI |
10.18653/v1/2026.evaleval-1 |
| ISBN |
9798891764293 |
| Full Text |
|
| Is Part of |
EvalEval 2026: proceedings of the Workshop on Evaluating Evaluations, July 4, 2026 / Mubashara Akhtar, Jan Batzner, Leshem Choshen, Avijit Ghosh, Usman Gohar, Jennifer Mickel, Ichhya Pant, Zeerak Talat, Michelle Lin (editors).. Kerrville : Association for Computational Linguistics, 2026. p. 128-160.. ISBN 9798891764293 |
| Keywords [eng] |
benchmarks ; evaluations ; AI evaluation ; validity ; AI safety |
| Abstract [eng] |
Effective AI risk assessment relies on the quality of evaluations. Currently, there are large quality differences, such as in construct validity and annotation, between existing benchmarks. In this work, we propose a quality scorecard for benchmarks designed to make this diversity easier to navigate. The scorecard employs two main components: dimensions, which provide granular scores of an evaluation under that dimension, and classifications, which correspond to concrete use-cases ranging from research to post-deployment. By establishing a common language and objective methods, this framework aims to aid in transparency and raise the baseline quality of benchmarks used across the ecosystem. |
| Published |
Kerrville : Association for Computational Linguistics, 2026 |
| Type |
Conference paper |
| Language |
English |
| Publication date |
2026 |
| CC license |
|
