SynthWorld baselines and benchmark demonstrations

These are deliberately naive reference baselines: each score illustrates what its benchmark measures, not the state of the art. Every number below is regenerated from the public-only baseline adapters by the command in Reproduce. All data is safely synthetic.

Reproduce

uv run python examples/generate_benchmarks_doc.py

make baselines checks this file for drift in CI.

Baseline results

Asteria Agentic v1 baselines

Both baselines consume only the public bundle. Always-deny shows why accuracy alone is misleading on a deny-heavy fixture; the current-state baseline shows why final audit state cannot replace historical replay.

Enterprise authorization baselines

Each of these consumes the shipped reference pack for its family and deliberately fails one dimension, so the score shows what the dimension detects. Only the metrics that separate the baselines are listed; every family publishes more, each with its own denominator and no aggregate.

The reference packs are conformance fixtures, not statistical benchmarks — denominators here are in the tens, and their evaluator answer keys ship in the contract packages. A perfect score is evidence that an adapter conforms, never that a system generalises.

Ambiguity v2 error floor

The v2 pack’s difficulty is computed, not claimed: its genie floor is the Bayes error of the generator itself - the accuracy of an optimal solver restricted to the modelled observation (the rendered values, the comparable structure and the true prevalence) and holding the public law. Read the pack as a hardness certificate, not a capability leaderboard: the ceiling 1 - floor is the most any system can achieve, and transcribing the published rule already reaches it, so the informative number is a resolver’s gap to the genie. A score above the ceiling is exploiting signal the model says should not exist; a score within the genie’s confidence interval is, statistically, at ceiling.

• Published floor: 0.1098 (±0.0073, 95% Wilson interval)
• Ceiling 1 - floor: 0.8902
• Technique premium: 0.0728 (gate ≥ 0.05)
• Floor band: [0.08, 0.12]
• Estimated over 7030 pairs from 100 seeds
• Decision digest: f2c68dd5c7f9ed1d49d63af182ce339c

The digest binds these numbers to every decision-relevant constant; any parameter move invalidates them until examples/compute_ambiguity_floor.py is rerun.

Why SynthWorld, not a row generator

What the visuals show

A. One persona, conflicting public records

flowchart LR
    entity["One entity"]
    entity --> record0["conference: Katherin Oconor"]
    entity --> record1["social: Katherine O'Connor"]
    entity --> record2["directory: Katie Oconnor"]

One real person surfaces under three spellings across three sources; the answer key knows they are one entity.

B. Broker removal and reappearance timeline

flowchart LR
    state0["2026-01-22<br/>found"] --> state1["2026-01-27<br/>removal_requested"] --> state2["2026-02-26<br/>confirmed_removed"] --> state3["2026-04-12<br/>reappeared"]

A listing confirmed removed can reappear at a later virtual date; the benchmark plants this so removal-tracking systems can be tested.

C. Public input vs evaluator truth

flowchart TD
    public["Public corpus"] --> sut["System under test"]
    sut --> predictions["Predictions"]
    answers["Separately serialized answer key"] --> scorer["Scorer"]
    predictions --> scorer
    scorer --> results["Scored results"]

Products consume only the public projection; evaluators join the separately serialized truth to score.

Size and limits

• The benchmarks are frozen at seed 20260719, 10 personas (18 records for the adversarial entity-resolution pack).
• Asteria Agentic v1 is separately frozen at 24 events and 11 action attempts; it is a conformance fixture, not a statistical leaderboard.
• Baselines are intentionally simple and are NOT state of the art.
• Scores illustrate the benchmark’s discriminative power, not system quality.
• Numbers change only through a deliberate benchmark-version transition.

See DATA_DICTIONARY.md for field definitions and GOLDEN_REVIEW.md for the frozen benchmark review record.