AI Evaluation Consensus Statement

1. DesignStrong ConsensusBroad Consensus
   Stakeholders and roles are explained clearly
   Subitems

   • State who built and who ran the evaluation
   • State any other identified affected stakeholders for the outputs
   • Explain stakeholder/expert consultation that informed design, if any
   • Explain any normative assumptions about how/why the eval is used
2. DesignStrong ConsensusBroad Consensus
   The report (or preregistration) explains purpose, use context, decision linkage, audience, and relation to existing evaluations
   Subitems

   • Explain the purpose and, if relevant, success objective of the evaluation
   • Explain intended use context and lifecycle placement
   • Explicitly mention any decision linkage and actions planned
   • Explain audience and interpretive boundaries
   • Report the relationship to existing evaluations and benchmark choice rationale
3. DesignStrong ConsensusBroad Consensus
   The report provides a construct definition and scope
   Subitems

   • Explain measurement of the construct/capability and the target claim
   • Explain sub-areas/aspects of the construct, and accompanying reporting plan
   • Provide an explanation or map from construct to real-world use/applications
4. DesignStrong ConsensusBroad Consensus
   Explain the problem framing, threat, and/or consequence model
   Subitems

   • Specify capabilities, failure modes, or threat model
   • Discuss different performance levels and outcomes
5. DesignStrong ConsensusBroad Consensus
   Note any limits to validity and design trade-offs
   Subitems

   • Explain external validity gaps and expected limits
   • Explain representativeness and coverage, construct contamination, and item-format impacts
   • Note any internal/external validity trade-offs
6. DesignStrong ConsensusBroad Consensus
   Specify and explain metrics, aggregation, and interpretation
   Subitems

   • Explain the metric(s) used by the evaluation
   • Explain the score meaning, and the range, including floors or ceilings, and explain any baseline(s) or natural comparison
   • Explain aggregate and subgroup metrics, and asymmetry handling if relevant
7. DesignStrong ConsensusBroad Consensus
   Quantify uncertainty, robustness, and interpretive limits
   Subitems

   • Explain any sources of uncertainty/variation and trade-offs
   • Perform and report input variation and robustness (prompting/stochasticity)
   • Design to allow uncertainty quantification
   • Explain saturation, ceiling effects, and interpretability of above-human performance
8. DesignStrong Consensus
   Perform and report statistical analyses
   Subitems

   • Perform power analysis or other sample size justification, and report detectable effect sizes or equivalent
   • Address multiplicity control, preregistration, and researcher degrees of freedom
   • Explain distributional assumptions, dependence structure, and model misspecification risks
9. DesignStrong ConsensusBroad Consensus
   Explain task and item construction, and perform design validation and report
   Subitems

   • Explain the task/item development workflow, and the selection logic
   • Explain any relationship to prior evaluations/literature, and provide a rationale for any adaptation
   • Perform validation of task/item criteria, and mention any domain-expert consultation
10. DesignBroad Consensus
    Note item sourcing, provenance, representativeness, and modality assumptions
    Subitems

    • Explain legal and ethical sourcing and permissions
    • Explain item provenance, representativeness, and coverage of items
    • Explain formats, languages, and modalities (including multimodality) and validity and robustness implications
11. DesignStrong ConsensusBroad Consensus
    Report human-subjects ethics, oversight, and safeguards where relevant
    Subitems

    • Report any ethics/IRB approvals and oversight for human-involved components
    • Report participant safeguards and protocol transparency (when humans are involved)
12. Before ExecutionStrong ConsensusBroad Consensus
    Provide a correctness definition and explain ground-truth validity
    Subitems

    • Provide correctness definition and ground-truth specification
    • Report validation approach and treatment of malformed/refusal outputs
    • Check and report human reference performance, and explain comparability (if relevant)
13. Before ExecutionStrong ConsensusBroad Consensus
    Create and provide rubrics and judge-system specification
    Subitems

    • Create and provide rubric/scoring system, including construction and validation details
    • Report LLM judge(s) model/prompt details and validation
    • Report human judge(s) recruitment/demographics/compensation/consent details
14. Before ExecutionStrong ConsensusBroad Consensus
    The evaluation performs and explains judge training and quality control processes, then reports them
15. Before ExecutionStrong Consensus
    Perform contamination, gameability, and evaluation-awareness controls
    Subitems

    • Perform data contamination checks and exposure assessment
    • Gameability/sandbagging prevention and detection mechanisms
    • Perform evaluation-awareness checks (if relevant)
    • Report access/secrecy controls for sensitive evaluation components
16. ExecutionStrong Consensus
    As-run logging and reproducibility provenance
17. ExecutionBroad Consensus
    Document and report execution-time adaptation of the evaluation and elicitation controls
18. ExecutionStrong ConsensusBroad Consensus
    Document failure patterns, run differences, and execution constraints
19. LifecycleStrong ConsensusBroad Consensus
    Provide evaluation outputs or document artifact availability and access pathways
    Subitems

    • Licensing terms, open/modifiable availability, and model access requirements (black-box, weight access, or internal activations)
    • Provide results and replication data publicly, or explain any access alternatives if non-public
    • Provide reusable code or access conditions for third-party evaluations and future reuse
20. LifecycleBroad Consensus
    Document governance, versioning, and re-evaluation criteria (if any exist) for future use
    Subitems

    • Provide plans for updates, deprecation, and retirement, if relevant
    • Provide version/build status and change-tracking policy
    • Explain any criteria for valid future use on new models/systems
    • Monitoring triggers and re-evaluation protocol
21. LifecycleStrong ConsensusBroad Consensus
    Provide operational documentation, portability, and maintenance support
    Subitems

    • Include operational documents/APIs/requirements along with publication, and document any portability support
    • Provide guidelines for reuse, adaptation, and safe integration
    • The evaluation identifies maintenance ownership and feedback/contact channels, if any exist
22. Reporting & PublicationBroad Consensus
    Include resource accounting and operational trade-offs
    Subitems

    • Report compute/sample/resource requirements for running the evaluation
    • Include cost accounting methodology and reported cost metrics (tokens, dollars, time)
    • The report describes operational trade-offs across run modes and deployment contexts
    • Explain any difference between planned resource profile and realized execution or constraints encountered
23. Reporting & PublicationStrong ConsensusBroad Consensus
    Provide results interpretation and evidence for claims
    Subitems

    • Note any baselines and provide guidance for interpretation of results
    • Perform and report sensitivity analyses, note differences vs prior reports, if any
    • The report ties conclusions to results, relates them to decision criteria, and notes any limitations for comparisons
    • Note any disagreements or disputes over interpretation
24. Reporting & PublicationStrong Consensus
    Report multi-model comparison standards and trend claims, if relevant
    Subitems

    • The report standardizes comparisons across reports, models/versions, and different sources of variation or uncertainty
    • Include trend analysis and future-performance projections, if possible
    • Note model selection and setup uncertainty in comparative conclusions
25. Reporting & PublicationBroad Consensus
    Explain publication status and reporting artifacts
    Subitems

    • The report notes submission/publication status
    • Complete an evaluation card/checklist and machine-readable and human-readable reporting template
    • Include content warnings and communication safeguards (if relevant)
    • Provide a post-publication correction/update pathway
26. Reporting & PublicationBroad Consensus
    Mention evaluation usage rules or conditions, release context, and access constraints
    Subitems

    • Report any evaluation of Pre-release, pre-deployment, or checkpoint model versions and impacts on reporting validity
    • Provide a timeline of runs/results release and recipient audiences
    • Access/publication restrictions and 'file-drawer' risks, where evaluations are not reported
    • The report notes any GUID/canary usage, or other verifiable cryptographic commitment mechanisms (e.g. hashes of non-public preregistration)
    • Include a disclosure policy or explanation for sensitive information hazards
27. Reporting & PublicationStrong ConsensusBroad Consensus
    Provide a replication package and reproducibility guide, or explain reproducibility constraints
    Subitems

    • Make replication code and execution package available, or otherwise explain how reproducibility can be achieved
    • Ensure users can reproduce reported outputs or understand and audit irreproducible components
    • Note data retention plans and auditability of reproducibility artifacts

• AI Security Institute
• AVERI
• Carnegie Mellon University
• Center for Open Science
• Center for Security and Emerging Technology
• Centre for Governance of AI
• ETH Zurich
• European AI Office
• Google DeepMind
• Harvard University
• Hugging Face
• IBM Research
• Imperial College London
• Intel
• METR
• MIT CSAIL
• NYU
• OpenAI
• Oxford Internet Institute
• RAND Center on AI, Security, and Technology
• Schmidt Sciences
• Tsinghua University
• UC Berkeley

Partial list of affiliations represented during the Delphi process. Participation does not imply endorsement of the final results.