Contributors
Building an AI safety eval? Find out which open research question it informs, what evidence your evaluation already provides, and what would be needed to make your result independently verifiable. Early prototypes are welcome.
Three levels
| Level | You provide | Registry returns |
|---|---|---|
| Idea / design | A URL to a repo, benchmark, paper, or a short description | Relevant research questions and suggested next tests (via a maintainer reply on an issue) |
| Pilot / evidence | Cases, results, or a runnable harness exported as a sketch | A public gap analysis: coverage, qualification floors still missing, bar preview |
| Qualifying attempt | A frozen, reproducible evaluation that meets the contract's machine checks | Scoring against the frozen contract (YES / NO / OTHER at snapshot, once the registry is a resolution source) |
Can I contribute an incomplete eval? Yes. A sketch is visible on this site and never resolves a question. You only need to match a frozen contract when you aim for a qualifying attempt. Proposing a new question is optional and a larger commitment than filing evidence against one of the seventeen listed below.
Start with an idea
No repo or freeze yet? Open an issue with a link or description: Idea or existing eval. If nothing on the list fits: Eval does not match a listed question.
Run a pilot
The workbench has scaffolds for Markets 1, 4, 8, 11, and 12 (Inspect for 1 and 4; custom wraps for 8, 11, and 12). Useworkbench new, freeze, run, and export for those, orworkbench new --market market-NN --type custom for any other frozen contract. If you already have results, start a registry sketch with python -m validator new instead. See sketches/README.md and AGENTS.md for the step-by-step process agents and contributors follow.
Published sketches: sketch list (sorted by completeness).
Worked example: Dietz edge cases (Market 4)
A partial Market 4 pilot wraps published examination responses (no model calls), exports to zarncke-2026-10-07-dietz-edge-cases. The dry run shows coverage and which qualification floors are still missing (episode count, adversarial route, and more). That gap analysis is the immediate value; extending the sketch or leaving it as a public record are both valid next steps.
Current completeness: 20%. Status: not ready to submit.
Runner (excerpt; full file on GitHub): contrib/zarncke-2026-10-07-dietz-edge-cases/run.py
"""Copy Dietz's published gold-example responses into a raw log. Does not call any model."""
...
for case in cases:
cid = case["case_id"]
published = json.loads((HERE / case["source"]).read_text(encoding="utf-8"))
rec = {
"case_id": cid,
"later_behavior_probe_published": False,
...
}
f.write(json.dumps(rec, ensure_ascii=False) + "\n")While the host is not independent, attempts submitted or authored by aintelope or Gunnar Zarncke are not accepted; sketches may still be merged. See GOVERNANCE.md.
Seventeen questions
Pick the contract your eval informs, then file a sketch or issue against it.