SynthWorld

Connected synthetic identity worlds for testing identity, privacy, and access systems.

Generate fictional identities and relationships for your tests, or exercise agent access decisions across delegation, credential, and lifecycle changes.

Choose your starting point

Generate synthetic identity data	Test agent authorisation
Create repeatable connected fixtures for matching, privacy, and graph workflows.	Inspect concrete access failures and compare teaching policies before connecting your system.
Identity quickstart	Agent demo and report

Generate connected synthetic identities with 0.18.0; no policy engine is needed:

pip install idcognito-synthworld==0.18.0
synthworld generate --seed 20260719 --persona-count 10 --output world.json

The packaged agent demo is new in 0.18.0. With the package installed, run the complete teaching exercise:

synthworld-demo run --output authority-demo

Open authority-demo/results/policy-comparison.html. Rerun in a new directory with --omit-delegation-check to omit the combined ReBAC authority view and allow a post-revocation action. This uses local teaching policies, not a deployed IAM system.

The --omit-delegation-check teaching flag omits the combined policy’s entire ReBAC authority view, including active delegation, relationship, and coverage checks; RBAC and ABAC remain enabled. The manifest setting is a declaration, not execution attestation. Scoring accepts externally edited, correctly bound traces and evaluates their decisions rather than rerunning the teaching policy. Retain the result manifest and referenced submission manifest alongside metric JSON to identify the declared mode and exact submitted bytes.

SynthWorld creates fictional data; it does not anonymise sensitive real records. For evaluations, give the system only the task’s documented public inputs and keep expected answers in the evaluator process.

Why SynthWorld

Requirement	SynthWorld approach
Repeatable evaluation	Explicit seeds/configuration, canonical ordering, frozen fixtures, and checksums
Connected test data	Coherent worlds rather than independent fake rows
Adversarial cases	Conflicts, ambiguity, lifecycle changes, and negative controls are planted deliberately
Controlled oracle exposure	Product-facing observations and evaluator truth use separate artifacts and contracts
Reproducible claims	Versioned schemas, scoring formulas, benchmark identities, and integrity metadata

A frozen conformance fixture is evidence that an adapter handles the declared cases; it is not automatically evidence of real-world transfer or a vendor leaderboard.

Current benchmark families

SynthWorld includes deterministic surfaces for connected identity fixtures, privacy and exposure, extraction, entity resolution, relationship inference, risk calibration, agent authority, and enterprise identity/access evaluation. These families do not all share the same maturity, publication state, CLI, or statistical meaning.

Use the generated benchmark catalogue for governed current state and the human-readable BENCHMARKS.md for benchmark context and reference results.

The core identity world is a smoke surface

The frozen core world is intentionally small and structurally simple. It is useful for deterministic fixtures, demonstrations, and CI, but it is not a transfer surface for claims about real populations. Use richer generated profiles when graph structure or population variation is part of the test.

See the identity-world guide and BENCHMARKS.md for measured limits.

What the ambiguity pack does and does not measure

The ambiguity families exercise conflicting evidence and evidence-aware resolution. The frozen reference pack is a conformance fixture with deliberately small slices; the generated v2 construction uses a different difficulty model. Neither should be presented as proof of real-world transfer simply because a system scores well.

See the identity-resolution guide and BENCHMARKS.md for the current constructions, baselines, and limitations.

Public input and evaluator truth

Only inputs explicitly documented as public belong on the product side. Do not assume that an artifact is product-safe merely because SynthWorld generated it.

For example, generate-public-extraction, generate-public-connections, and generate-risk-public emit product-facing projections. By contrast, generate-extraction and generate-connection-benchmark emit evaluator or annotated bundles containing expected answers and must not be passed to the system under test. Some benchmark commands write both public/ and evaluator/ subtrees; in that case, pass only the documented public subtree to the product or model.

Evaluator artifacts contain the information used to score the resulting prediction or trace.

public benchmark input
        |
        v
 system under test
        |
        v
 prediction / trace ---------+
                             |
 evaluator truth ------------+--> independent metrics

Physical separation prevents accidental oracle use; it does not make a published reference fixture secret.

Enterprise identity and access

The enterprise surface can compile operator-authored structure into a deterministic, safely fictional identity/access universe and provides bounded reference benchmark and projection surfaces around it. Public product inputs and canonical evaluator truth remain separate.

Start with the enterprise access guide, follow the enterprise authorization guide for the installed-package experiment path, and use the normative enterprise contract for the versioned artifact requirements.

What the enterprise surface does not claim

Importing enterprise structure is not anonymisation; authored structural inputs can remain sensitive.
Offline evaluation and standards-shaped projections are not deployed IAM or enforcement.
Published reference packs are conformance fixtures, not blind statistical benchmarks or vendor leaderboards.

Install

SynthWorld requires Python 3.12 or newer. The distribution is idcognito-synthworld; the import package and CLI are synthworld.

pip install idcognito-synthworld
synthworld generate --seed 20260719 --persona-count 10 --output world.json

The same explicit inputs reproduce the same deterministic fixture. Continue with Getting Started before using a benchmark scorer.

Evaluate a system

Every integration follows the same pattern: give the system only explicitly public input, normalize its native output into the task-specific prediction or trace contract, then score it against separately loaded evaluator truth.

See Evaluating a system for runnable examples and metric interpretation.

Validate before you score

Use a task validator where one exists. Structural validity means a submission can be scored; it does not mean the system performed well.

synthworld validate agentic-trace --predictions observed-actions.jsonl

Use Asteria Agentic v1

A minimal agent-authority evaluation flow is:

synthworld generate-agentic --output asteria-agentic-v1
synthworld validate agentic-trace --predictions observed-actions.jsonl
synthworld evaluate agentic --predictions observed-actions.jsonl --summary

Give only the generated public/ tree to the system under test. Keep the evaluator side out of the adapter path even though the frozen reference truth is publicly inspectable. See the full Asteria Agentic v1 guide.

For configurable generated enterprise-agentic worlds, explicitly select the generated profile. Smoke preserves its released V1 contract; standard and longitudinal use a separate V2 scale/lifecycle family:

synthworld generate-enterprise-agentic \
  --profile generated \
  --tier smoke \
  --seed 20260814 \
  --output generated-enterprise-agentic

An external adapter receives only generated-enterprise-agentic/public, replays its events in order, and writes the observations it actually obtained. Validate without evaluator access, then score in a separate evaluator process:

synthworld validate generated-enterprise-agentic-trace \
  --benchmark-root generated-enterprise-agentic \
  --predictions observed-actions.jsonl
synthworld evaluate generated-enterprise-agentic \
  --benchmark-root generated-enterprise-agentic \
  --predictions observed-actions.jsonl \
  --summary

This is a deterministic benchmark-data generator, not an IAM product, policy engine, agent framework, hosted simulator, or vendor leaderboard. A reference organisation topology can inform the supported count configuration; SynthWorld does not import its named entities or relationships. See the agent-authority guide for the replay, decision-only SUT, and provenance boundaries, and the scale-tier guide for configuration, lifecycle, metrics, and measured runtime/memory characteristics.

Verify every claim

For reproducible evaluation, retain the benchmark identity, relevant seed and configuration, schema/scoring versions, artifact checksums, and the exact prediction or trace bytes that were scored. Interpret each metric through its own denominator and support semantics rather than hiding weak dimensions behind an aggregate.

Use the data dictionary and benchmark inventory for authoritative contracts and reference results.

Roadmap and integrations

Current direction is summarized in the documentation roadmap and the repository ROADMAP.md. Governed registries—not issue state—remain authoritative for current capability and benchmark publication status.

Develop from source

uv sync --locked --all-groups
make ci

Contribution guidance is in CONTRIBUTING.md.