benchmarks

Heidi Feys: built-in AI evaluators are not a simple solution, benchmarks are not yet a science

Irene Solaiman, head of policy at Hugging Face, said at TechCrunch Disrupt that AI model evaluation methods have not yet matured into a coherent science: results are hard to reproduce and compare with each other. The idea of built-in third-party evaluators, promoted by labs as an alternative to government regulation, requires standardization; Anthropic's decision to hire Accenture drew criticism.