Skip to content
SynthWorldDeterministic identity benchmarks
Esc
navigateopen⌘Jpreview
On this page

Generate enterprise-agentic scale tiers

Available since 0.17.0. Standard and longitudinal use their own V2 contracts; the released smoke V1 contract remains unchanged.

SynthWorld provides three generated enterprise-agentic tiers without changing the frozen Asteria Agentic v1 benchmark or the released generated smoke V1 contract. These tiers measure generated workload and control behaviour. They are not a vendor leaderboard, production authorisation claim, IAM service, or agent runtime.

Tier and contract ladder

Tier Contract Default topology Default action cases Purpose
smoke enterprise-agentic-generated-1.0.0 1 organisation, 25 humans, 5 agents, 8 runtimes, 6 resources 7 Fast CI and adapter validation
standard enterprise-agentic-generated-2.0.0 2 organisations, 250 humans, 36 agents, 72 runtimes, 36 resources 23 Ordinary comparative evaluation and multi-tenant controls
longitudinal enterprise-agentic-generated-2.0.0 The standard topology over 180 virtual days 31 Rotation, joiner/mover/leaver, suspension, policy change, offboarding, revocation propagation, evidence loss, and audit-time evaluation

stress is explicitly deferred to the generic scale work in issue #3. Large generated worlds remain out-of-band workloads and never become mandatory package fixtures.

Generate and consume a tier

synthworld generate-enterprise-agentic \
  --profile generated \
  --tier standard \
  --seed 20260821 \
  --output generated-standard

Use --tier longitudinal for the lifecycle schedule. Use --public-only when the machine preparing product input must not write an evaluator tree. A JSON file passed with --config supplies a complete validated configuration; explicit --tier and --seed values are authoritative and become part of the resolved benchmark identity.

The Python API preserves smoke V1 and exposes V2 separately:

from pathlib import Path

from synthworld.agentic.enterprise import (
    EnterpriseAgenticScaleTierV2,
    default_enterprise_agentic_generation_config_v2,
    export_generated_enterprise_agentic_scale_benchmark,
    generate_enterprise_agentic_scale_world,
)

config = default_enterprise_agentic_generation_config_v2(
    EnterpriseAgenticScaleTierV2.STANDARD,
    seed=17,
)
generated = generate_enterprise_agentic_scale_world(config)
export_generated_enterprise_agentic_scale_benchmark(Path("generated-standard"), generated)

Give only generated-standard/public/ to an adapter. The public tree contains the base agentic snapshot and ordered events, team, population, resource, and opaque credential-handle projections, public lifecycle events, the scenario, and the tool schema. It contains no canonical bindings, expected decisions, case labels, authority truth, or integrity answer key. The evaluator tree cross-binds the complete public artifact-set digest.

The existing observed-action JSONL boundary works for every tier:

synthworld validate generated-enterprise-agentic-trace \
  --benchmark-root generated-standard \
  --predictions observed-actions.jsonl
synthworld evaluate generated-enterprise-agentic \
  --benchmark-root generated-standard \
  --predictions observed-actions.jsonl \
  --summary

Configuration surface

EnterpriseAgenticGenerationConfigV2 is immutable, rejects extra fields, and binds the profile, schema, generator, canonical-serialisation and event-schedule versions, seed, tier, and every resolved knob. Its bounded submodels configure:

  • organisations, departments, teams, employees, contractors, suppliers, external partners, agents, runtimes, and resources;
  • direct-human, organisation, and agent sub-delegation ratios, delegation depth, capability breadth, and scope density;
  • credential validity, shared-identity prevalence, and runtime-binding breadth;
  • independent counts for authorised, capability, child-delegation, runtime, shared-credential, tenant, revocation, later-grant, evidence, attribution, credential, policy, rotation, suspension, offboarding, and propagation cases;
  • virtual duration, rotation interval, evidence retention, policy change, and agent offboarding; and
  • explicit principal, event, and case generation limits.

Invalid ratios, impossible topologies, lifecycle controls on standard, missing lifecycle controls on longitudinal, cross-tenant cases with one organisation, and limit overruns fail before artifact export.

Longitudinal event boundary

The base agentic 1.0.0 event union remains unchanged. V2 serialises lifecycle events as a separate, typed lifecycle-events.json stream and keeps their ordering stable with sequence_index plus UTC timestamps. Rotation, suspension, offboarding, policy activation, principal join/move/leave, and revocation propagation compile to base credential-validity, delegation, policy, action, evidence, and audit records so the hardened base replay and evaluator still derive action-time and audit-time truth. The evaluator-only lifecycle case map supplies the more specific V2 cohort labels.

Determinism, integrity, and metrics

Dedicated UUID5 namespaces derive opaque IDs only from the explicit configuration identity. Collections use canonical ordering; event streams retain semantic order; JSON uses UTF-8, LF, and one trailing newline. Complete loading rebuilds the hardened base benchmark, re-derives metrics, regenerates the declared world, and compares the exact public and evaluator bytes.

Metrics are derived from generated records rather than copied from requested counts. Every count and distribution states its denominator. Reports include topology and population counts, graph components, owner/delegation depth, delegation branching, runtimes per agent, credential-runtime binding breadth, allowed/denied actions, case prevalence, isolated controls, revocations, evidence loss, lifecycle kinds, and referential and canonical-binding integrity.

Runtime and memory receipt

Host observations are deliberately outside deterministic artifacts. Reproduce the checked-in receipt with:

uv run python tools/measure_enterprise_agentic_scale.py \
  --source-revision <commit-or-source-digest> \
  --dependency-lock uv.lock \
  --iterations 3 \
  --output enterprise-agentic-tier-performance.json

The repository receipt records Python 3.12.12 on Linux with the locked dependency digest. Three-iteration medians were:

Tier Generate Serialise Replay Score Peak traced memory
standard 2.894 s 0.035 s 0.056 s 0.048 s 5,465,472 bytes
longitudinal 4.043 s 0.037 s 0.061 s 0.048 s 5,480,891 bytes

These are environment-specific observations, not deterministic promises. Each row binds its resolved configuration and complete public artifact-set digest in enterprise-agentic-tier-performance.json.

Was this page helpful?