Finalizing Results

Building Consensus on AI Evaluation Practices

A cross-sector initiative to establish community-endorsed guidance for conducting and reporting high-quality AI evaluations.

The Delphi rounds are complete and results are being finalized.

About The Process Browse Item Explorer Follow results →

Summary

AI evaluations increasingly inform safety, governance, and deployment decisions—yet current practices remain uneven, siloed, and difficult to compare. We organized a broad, cross-sector Delphi panel to establish community-endorsed, concise guidance around practices for conducting and reporting high-quality evaluations, and are now finalizing the results. Such broadly endorsed guidance will create a shared reference and hopefully improve evaluation quality across labs, auditors, researchers, ethics teams, downstream model users, and practitioners. For more details, see our published call to join the process in Patterns.

Who is involved

Consortium & Participants

This study is based on a consortium model, with organizational support and input from the organizations endorsing the project, and with contributions from many advisors — especially the authors of the protocol, Daniel Stuart Schiff, Michael Noetel, Stephen Casper, and David Manheim; the consortium manager, Irving Torres; and the study team, Andrea Loehr, David Manheim, Yixiong Hao, Drishti Sharma, Ishan Khire, Yilin Huang, Nidhi Sakpal, and Paulo Bova, among others.

Endorsing Organizations1

AI Standards Lab

Participating Individuals From2

  • AVERI
  • Centre for Governance of AI
  • Center for Open Science
  • Center for Security and Emerging Technology
  • Carnegie Mellon University
  • ETH Zurich
  • Imperial College London
  • Google DeepMind
  • Harvard University
  • Hugging Face
  • IBM Research
  • ILINA Program
  • Intel
  • METR
  • MIT CSAIL
  • OpenAI
  • Oxford Internet Institute
  • RAND Center on AI, Security, and Technology
  • Schmidt Sciences
  • Tsinghua University
  • UC Berkeley
  • NYU
  • AI Security Institute
  • European AI Office

1) Organizational endorsement does not imply agreement with the final results; it is instead an endorsement of the importance and legitimacy of the process laid out in the protocol.
2) This is a partial list of affiliations represented during the Delphi process.

The Challenge

Why This Matters Now

Evaluations are increasingly relied upon across labs, auditors, regulators, and researchers, but shared expectations about what "well-done" looks like remain thin.

  • Many evaluations lack documentation on design choices
  • Statistical soundness and uncertainty often underspecified
  • Inconsistent scoring and applicability criteria
  • Results difficult to compare across organizations
  • Missing basics lead to avoidable failures
📋

Not a Framework

We're not creating a new evaluation framework, benchmark, or metric—just making existing agreement visible.

⚖️

Not a Mandate

Not a governance mandate, compliance regime, or audit protocol. Guidance, not requirements.

🤝

Building Consensus

Not an attempt to resolve deep methodological disputes—focusing on areas of practical agreement.

Our Approach

What We're Building

A preregistered, multi-sector Delphi process to establish concise, community-endorsed guidance for conducting and reporting high-quality AI evaluations analogous to CONSORT, PRISMA, or TRIPOD.

📑

Preregistered Protocol (Published)

A publicly available, preregistered methodology specifying consensus thresholds, reporting rules, subgroup analyses, and deviation logging.
✓

Evaluation Checklist

A concise 1–2 page guideline form for auditors, labs, researchers, and other producers of AI evaluations to explain adopted or non-adopted practices, based on Delphi outcomes.

🏛️

Consensus Statement

A statement reflecting the importance of adopting practices with cross-sector agreement, inviting endorsement from academia, industry, standards bodies, civil society, and evaluation organizations.
Methodology

The Process

A structured, multi-round expert survey designed to identify where agreement already exists on concrete practices.

Systematic Review (Completed)

Broad review of evaluation frameworks, reporting checklists, measurement standards, and methodological critiques across ML, medicine, metrology, software testing, and experimental sciences.

Item Development(Completed)

Candidate items extracted from the systematic review, coded and mapped using a best-fit framework approach. Iteratively refined with expert feedback. Browse the Item Explorer.

Multi-Sector Participation (Completed)

Stakeholders across AI labs, auditors, academia, industry users, standards bodies, and government institutions contributed ratings and feedback to ensure broad representation in the consensus process.

Consensus Delphi Process (Completed)

Participants reviewed and rated proposed practices in areas they felt qualified to assess across the Delphi rounds.

Consensus Review (Finalizing)

Final synthesis is underway with preregistered constraints, transparent reporting of results including subgroup differences, and no post-hoc renegotiation.

Launch and Build Support (Upcoming)

Stakeholders will be invited to review and endorse the resulting consensus guidance, strengthening shared expectations for high-quality AI evaluations across the ecosystem.

Draft Statement

The Consensus Statement

AI evaluations provide important information for developers, deployers, and end users of AI systems. For this reason, it is important that they are built well and follow best practices. Therefore, based on the consensus of stakeholders and experts, we recommend the consideration and adoption of the below practices for constructing, documenting, maintaining, using, and reporting the results of evaluations.

Get Involved

Get Results Alert

Sign up to be alerted when the final results are available for review and endorsement.

For Individuals

✓
Get the results notice

Receive an alert when the final Delphi results and consensus materials are available.

✓
Review or endorse

Be notified when endorsement of the final outputs opens after synthesis and review are complete.

✓
Share the guidance

Help circulate and apply the final guidance once the report is released.

For Organizations

✓
Prepare to endorse

Receive a notice when organizations can review and endorse the final consensus outputs.

✓
Coordinate internal review

Use the alert to route the final materials to relevant policy, safety, governance, or evaluation teams.

✓
Support dissemination

Help share and maintain the checklist after publication.

Contact

Join the Conversation

The Delphi rounds are complete and results are being finalized. Reach out to learn more.

✉ [email protected]