Enterprise authorization with Topaz

This experiment series asks whether an external authorization system can consume released SynthWorld enterprise artifacts and reproduce the relevant authorization behaviour. Topaz is the first system under test. It is not part of SynthWorld’s oracle, and the results are not a vendor comparison or general authorization correctness claim.

The concrete Phase 2 question was:

Can an authorization system consume only released SynthWorld public artifacts, project them into its own policy model, and reproduce the scoreable authorization behaviour without using evaluator truth?

Phase 1 answered the narrower question of whether released relationship data could be transported into and queried from a Topaz directory. Phase 2 exercised a live authorization policy and the released mechanism scorers. Phase 3 used the 0.16.0 contracts to enforce the evaluator mount boundary, score composed decisions and exercise discriminating adversarial cases. None of the phases establishes general Topaz or authorization correctness.

Experiment boundary

The experiment kept generation and scoring on the SynthWorld side of the boundary and treated the Topaz adapter and projected policy as experiment-owned code:

fictional organization topology
    -> SynthWorld compilation
    -> public benchmark artifacts
    -> experiment-owned Topaz projection and policy
    -> Topaz authorization requests
    -> sealed predictions
    -> separate SynthWorld evaluator scoring

• SynthWorld owns deterministic generation, public benchmark contracts, evaluator truth and mechanism scorers.
• The experiment owns the organization-topology mapping, Topaz data projection, Topaz configuration and Rego policy.
• Topaz is the system under test. Evaluator artifacts are not Topaz inputs.
• The organization topology is an experiment input, not a SynthWorld schema.

Evidence status

The Phase 2 source-only baseline is locally tagged phase2-baseline-0.15.0. A clean local clone completed all 32 validation checks and reproduced submission digest 89099e3b55226cd6bd378f6dc7a2153aed3ee8d0e6e7fe3f9781d5be25a69f05. The experiment Git tag is not hosted as a separately fetchable remote. The release archives below preserve the source and retained evidence, making the record independently downloadable but not yet independently reproduced.

Phase 1: directory feasibility

Phase 1 mapped a fictional enterprise topology into the released enterprise identity/access authoring schema, compiled it, loaded the public relationship data into Topaz, and ran 6,880 directory checks. It established that the public artifact could drive a real relationship directory and that evaluator bindings could remain outside the runner container.

It did not exercise Topaz’s authorization decision API or a composed policy. Its permissions were one-to-one aliases of relations written by the same build, so the perfect result was a directory round-trip rather than an independent authorization test. The source Compose file selected ghcr.io/aserto-dev/topaz:latest; the cached image was later identified as Topaz 0.33.16 with digest sha256:835868c04bdd7129127ea43642ffff7363d0bd26d5e1a37631fa881431054360, but that recovery does not make the original source fully pinned. The directory was not source-controlled, and its HTML pages were experiment-owned viewers rather than a released SynthWorld renderer.

Treat Phase 1 as historical feasibility evidence only. Its publication package must preserve the original prompt, Compose file, scripts, inputs and reports alongside these qualifications.

Phase 2: frozen local run

Run identity

The known working host was Fedora 44 on x86_64, kernel 7.0.10, with Docker 29.5.2, Docker Compose 5.1.4, uv 0.11.17 and Python 3.13. These host details are observational provenance, not inputs to deterministic SynthWorld generation. A published reproduction must capture them in a machine-readable run receipt rather than treating this host as the only supported environment.

Inputs and provenance

Both phases used the byte-identical britannia_global_bank_topology.yaml. The source is the redoubtlabs/redoubtia-agents repository at commit 5e59e7d9b311e55c321d105a80a034c23dd704bf. The original Phase 1 prompt shortened the name to britannia_global_topology.yaml; the filename used by both runs and the SHA-256 above are authoritative.

The topology describes a fictional bank but contains references to real technology vendors and a real-bank comparison. A published copy must carry a fictional and non-endorsement notice. Editing those names would create a different input and must produce a new experiment version and digest rather than silently replacing the recorded file.

The reproducibility inputs are more than the topology alone:

• the original Phase 1 prompt and its rendering amendment;
• the original Phase 2 build prompt and the later correction/review prompt;
• the topology, experiment configuration and namespace salt digest;
• Compose files, Topaz model manifest and Rego policy;
• mapping, projection, execution, sealing, scoring and validation scripts; and
• the hash-locked Python dependency set.

Prompts are process provenance: they explain how the experiment implementation was created, but they are not runtime inputs or normative benchmark contracts. The full agent transcripts are not required for reproduction and should not be published in place of curated prompt and review records.

Method

The Phase 2 runner performed seven stages:

1. It staged, hashed, mapped and classified every organization-topology field.
2. It generated the deterministic SynthWorld world and physically separate public and evaluator projections.
3. It projected only public artifacts into Topaz objects, relations, requests, a model manifest and Rego policy.
4. It loaded Topaz, read the directory data back and issued one authorization request for every public evaluation cell.
5. It normalized Topaz responses into three mechanism submissions and sealed their combined digest before scoring.
6. It opened the evaluator tree, verified the seal and ran the released scorers.
7. It checked input and output digests, counts, public/evaluator inventories, required files, submission coverage and presentation leakage.

The runner was repository-independent: it installed the released distribution and did not use a local SynthWorld checkout or editable installation. It was not documentation-only because implementation work still required inspection of the released package’s public Python surface.

Results and evidence

Every result below states its denominator. The evidence paths are the locations to retain inside the Phase 2 reference-run archive.

Topaz’s unscored composed output contained 2,762 allows and 447 denies. It downgraded 245 effective allows: 18 at the RBAC binding/lifecycle gate and 227 at the public ABAC guard. These are observed decision counts, not composed-decision accuracy. The released scorers deliberately emitted no aggregate score because the mechanism families have different denominators.

What the run supports

• The released package could generate the input and evaluator artifacts without a SynthWorld repository checkout.
• An experiment-owned adapter could project the public artifacts into a live Topaz directory and policy.
• Topaz returned a normalized answer for every public request without a transport error or fallback.
• The sealed predictions reproduced the scoreable mechanism decisions listed above.

What the run does not support

• Independent reproduction on another machine or from a published remote.
• A general claim about Topaz, policy-engine or authorization correctness.
• Accuracy of the composed Topaz decision.
• A discriminating test of principal binding.
• Enforced isolation of evaluator truth from the system under test.
• Portability of the Britannia-specific topology mapper to arbitrary organization YAML files.

Phase 3: frozen isolated reference experiment

Phase 3 is published as the immutable, GitHub-attested release enterprise-authorization-topaz-0.16.0-1. It is a frozen and unsupported evidence record, not a maintained adapter repository. Its release metadata binds the archive digests to the exact experiment source commit and tree.

Run identity

Enforced capability boundary

The benchmark-owner container generated separately typed public and evaluator artifacts. Projector and runner capabilities received public paths but no evaluator mount or evaluator credential. The runner recorded raw Topaz responses, normalized them into typed submissions and sealed the complete public, policy, adapter, package, product and result evidence. Only the offline scorer received the evaluator tree, and it verified every seal binding before loading truth.

A deliberate regression mounted the evaluator tree into a runner capability. The isolation probe rejected that process. This protects against accidental leakage through the supported Compose workflow; it is not an anti-cheating claim against a host administrator or Docker-daemon operator.

Results and controls

The exercised composed effective/final, RBAC, ABAC, conflict, lifecycle, runtime-gate and adversarial dimensions each scored 1.0 at their stated denominators. The report computes no aggregate. It separately reports empty or unexercised ReBAC dimensions, evaluator-keyed fields that a public consumer cannot submit and world properties that are not system scores.

The reproduction ZIP was executed from a fresh extraction before publication and passed all 40 checks. The deterministic archives were also built twice and compared byte-for-byte. Operational timing is recorded but excluded from deterministic submission identity.

Reproduction materials

Historical Phase 1 and Phase 2 assets

The SynthWorld 0.16.0 GitHub release publishes the following retained experiment assets. The ZIPs were built from the Phase 1 and Phase 2 work conducted against SynthWorld 0.15.0; they are historical evidence, not regenerated 0.16.0 results.

The v0.16.0 release predates repository release immutability and is not retroactively protected by that setting. Verify its downloads against the recorded checksums. A GitHub-generated source archive is not sufficient because the Phase 2 repository intentionally ignored generated evidence, so those files would be absent.

Phase 3 immutable assets

The Phase 3 release is immutable. GitHub locks its tag and assets and publishes a release attestation. gh release verify enterprise-authorization-topaz-0.16.0-1 verifies that attestation; gh release verify-asset verifies an individual local download.

Each Phase 3 archive remains understandable after it is detached from this page. Its root contains:

README.md
EXPERIMENT-METADATA.json
FILE-MANIFEST.json
SHA256SUMS
LICENSE
docs/
01-source/
02-synthworld-public/   # reference-run archive only
03-topaz-input/         # reference-run archive only
04-topaz-results/       # reference-run archive only
05-submission/          # reference-run archive only
06-evaluator/           # reference-run archive only; physically separate
07-reports/             # reference-run archive only
viz/                    # reference-run archive only

The embedded README repeats the evidence status, exact versions, prerequisites, commands, directory map, supported claims and limitations. FILE-MANIFEST.json classifies each payload file as experiment source, public input, system input, system output, sealed submission, evaluator evidence, evaluation report or public presentation.

The shortest verified reproduction path is:

sha256sum -c SHA256SUMS
unzip enterprise-authorization-topaz-reproduction-kit-0.16.0-1.zip
cd enterprise-authorization-topaz-reproduction-kit-0.16.0-1
bin/run_lab.sh

The embedded README records the dependency and isolation boundaries. Topaz has no published host port. First execution needs network access to fetch digest-pinned images and hash-pinned packages; reruns can operate from the retained local cache.

Reproducing versus adapting

An exact reproduction keeps the topology bytes, seed, configuration, policy, dependency lock and container digest unchanged. Its result is comparable only after the input, public-artifact, submission and report digests have been checked.

A related experiment must receive a new identifier and must not overwrite a frozen baseline. Its report should describe every changed input, mapping decision and policy assumption; regenerate all checksums; construct discriminating positive and negative cases; keep public inputs and evaluator truth separately typed and serialized; and report every metric with its denominator.

The Britannia mapper is organization-specific. A new organization can either extend that mapping explicitly or begin from the released synthworld scaffold-enterprise-access workflow. The adaptation guide in the reproduction kit should identify which files are SynthWorld-generated and which are experiment-owned, then explain how to build public requests, run the system under test, seal a blinded submission and score it separately.

An admission-only versus continued-runtime-authorization study inspired by changing providers, credentials, destinations, delegated authority or capabilities should be a new experiment series. It should define the protected execution transitions and mutations explicitly rather than retrofitting or silently redefining a frozen run.

Material limitations found by Phase 2

The composed decision was not scored

Topaz produced a composed decision that combined RBAC-family results with the ABAC guard. SynthWorld 0.15.0 had separate directory/RBAC, ABAC and ReBAC scorers but no submission or evaluator for the composed CompiledEnterpriseAccessStateV1 decision. Cross-tenant and scope-exceeded actions were denied by the experiment’s composition while their released directory/RBAC truth remained allow.

Later contract work does not retroactively change what the frozen Phase 2 run scored. A successor experiment must use a released composed-decision contract and record that new package version.

The binding cohort did not test binding

The public input exposed the observed account binding but not enough evidence to resolve its canonical subject. The reference policy therefore treated the binding gate as passing. All 15 wrong-binding cases were already denied because the RBAC derivation found no path, so ignoring the binding mechanism did not change the outcome.

Some submitted metrics were not publicly solvable

Several released prediction fields depended on evaluator-only identifiers or unpublished policy intent, including ABAC truth identifiers, RBAC derivation-path identifiers and separation-of-duty constraints. Phase 2 reported these metrics as not publicly winnable rather than folding them into a headline result.

Public policy could reveal the negative cohort

The ABAC vocabulary did not provide the tenant inequality needed by the experiment. The projected rule therefore enumerated the public cell identifiers in its cross-tenant scope. A separately derived tenant comparison agreed with all 63 cases, but it was a cross-check rather than the decision path.

Isolation was auditable, not enforced

Public and evaluator artifacts were physically separate and submissions were sealed before scoring, but evaluator truth existed on the same host while the system under test ran. The supported process did not read it; the filesystem did not make such a read impossible. Phase 3 corrected that limitation by running the system under test without an evaluator mount and giving evaluator access only to a separate scorer.

Phase 3 contract delivery

The earlier experiment produced four focused requirements. SynthWorld 0.16.0 and the frozen Phase 3 experiment completed them without adding Topaz runtime behaviour to the core package:

• #137 added a publicly constructible composed-decision submission and independent scoring after the frozen Phase 2 run.
• #138 added discriminating tenant, scope, binding, temporal, clearance and composed authority counterfactuals with hidden pair labels and explicit discriminating denominators.
• #139 added the released consumer API, digest helpers and supported end-to-end workflow after the frozen Phase 2 run.
• #140 delivered the isolated experiment as immutable release evidence rather than creating an ongoing adapter-lab maintenance commitment.

The experiment implementation is frozen. Future adapters and experiments remain owned by their authors and consume SynthWorld through its public contracts.

Community result submissions

The Experiment results Discussion category indexes community-authored experiments. It is not a public leaderboard or an evidence store. Each entry is labelled experiment:self-reported by default and must identify exact versions, inputs, denominators, durable artifact links, SHA-256 digests, the public/evaluator boundary and limitations.

Discussion posts, links and attachments are mutable and therefore non-authoritative. Authors own and support their experiments. SynthWorld maintainers do not promise to inspect, reproduce, repair, host, rank or endorse submissions. A maintainer may voluntarily apply experiment:schema-checked after structural validation or experiment:maintainer-reproduced after reproducing the exact sealed artifact; neither status is a product certification.