VariantGrid experimentation system

System snapshot

Experiment workflowTyped states · constraints · versioned assignment · retry-safe exposure logging

8 regression tests cover the connected workflow.

Explore: Build a test · Try assignment · Run safety tests
Generated record: VG-1.json

Assignment consistency100% sticky · 3M simulated assignments · 0.0402 pp max drift

Three allocation policies with 1,000,000 assignments each; 1.458 µs median sampled p95 across five local trials.

Explore: Try assignment
Generated record: VG-2R.json

Inference and event integrity5.13% false positives · 94.87% interval coverage · 5/5 safety scenarios passed

10,000 fixed-seed A/A simulations plus duplicate, missing, late, reordered, and cross-version event scenarios.

Explore: Run safety tests
Generated record: VG-3.json

Methodology

Reproducibility

benchmarks/results.json
artifacts/release_audit.json
artifacts/test-results.txt

Run locally: PYTHONPATH=src python3 scripts/verify_release.py

Source fingerprint: 789486cbfabb069087067d7b60826a5e85e97873284d56f150a484a74f90baa8

Measured scope

Assignment evidence uses deterministic local simulation for 50/50, 80/20, and 90/10 policies. Latency is an in-process Python microbenchmark sampled every 100 calls on CPU model not reported / CPython 3.14.7; it is not network latency.

Calibration evidence uses frozen seeds and synthetic A/A ground truth. Event integrity uses local SQLite tests. These results do not establish production durability, Postgres behavior, uptime, customer adoption, causal business impact, or revenue lift.

Calibration thresholds: 4.5–5.5% false positives and 94.0–96.0% coverage. Integrity categories: cross-version, duplicate, late, missing, reordered.

Assignment

Same user + version = same state.

starter_type=blank|guided_tour=False

Same twice

Probability: 50.0%
Version: v1

Why it repeats

PASS · repeat assignment identical

Assignment inputs

Typed values: { "starter_type": "blank", "guided_tour": false }
Policy: sha256-v1 · fixed-weight-v1
Hash bucket: 0.40278068 in [0.000000, 0.500000)

Version isolation

PASS · version namespace isolated

v1 → v2 · bucket 0.65122532 · state starter_type=template|guided_tour=False

Version is part of the hash namespace. A version change creates an independent deterministic draw; landing in the same state is valid and is not evidence of leakage.

Benchmark

Policy	Assignments	Repeat stickiness	Max drift	Observed counts
50/50	1,000,000	100%	0.0402 pp	`{ "control": 499598, "treatment": 500402 }`
80/20	1,000,000	100%	0.0027 pp	`{ "control": 800027, "treatment": 199973 }`
90/10	1,000,000	100%	0.0176 pp	`{ "control": 899824, "treatment": 100176 }`

Latency: 1.458 µs median sampled p95 across 5 local in-process trials; range 1.417–1.625 µs. This is hardware- and load-sensitive microbenchmark evidence, not API latency.

Inspect all latency trials

Trial	Assignments	Sample every	Sampled p50	Sampled p95	Sampled p99	Throughput
1	1,000,000	100	1.334 µs	1.417 µs	1.459 µs	712,775/s
2	1,000,000	100	1.375 µs	1.625 µs	2.166 µs	669,115/s
3	1,000,000	100	1.375 µs	1.458 µs	1.500 µs	712,365/s
4	1,000,000	100	1.375 µs	1.458 µs	1.500 µs	714,856/s
5	1,000,000	100	1.375 µs	1.458 µs	1.500 µs	715,581/s

Event integrity

5/5 passed

Duplicate

Assignment, exposure, and outcome remain distinct; retrying an identical exposure or outcome cannot change the exposed or converted counts.

Test: same stable exposure and goal IDs delivered twice

Observed pipeline result

{
  "accepted_deliveries": 3,
  "duplicate_deliveries": 2,
  "delivery_acceptance": [
    true,
    true,
    false,
    true,
    false
  ],
  "stored_event_type_counts": {
    "assignment": 1,
    "exposure": 1,
    "goal": 1
  },
  "binary_counts": {
    "control": [
      1,
      1
    ]
  },
  "converter_count": 1
}

Reproduce: PYTHONPATH=src python3 -m unittest tests.test_integrity_lab.IntegrityLabTests.test_duplicate_row_uses_real_idempotent_ingestion -v

Missing Exposure

Assignment does not count as exposure; an outcome without qualifying exposure must be excluded from the default readout.

Test: assignment and goal exist, but no exposure was recorded

Observed pipeline result

{
  "accepted_deliveries": 2,
  "assignment_count": 1,
  "exposure_count": 0,
  "goal_count": 1,
  "binary_counts": {},
  "converter_count": 0,
  "outcome_excluded": true
}

Reproduce: PYTHONPATH=src python3 -m unittest tests.test_integrity_lab.IntegrityLabTests.test_missing_exposure_row_excludes_assignment_only_subject -v

Late

Late delivery must reconcile using occurred_at; receipt order cannot erase a valid same-version exposure-to-outcome sequence.

Test: exposure arrives after its outcome but has an earlier causal event time

Observed pipeline result

{
  "receipt_order_event_ids": [
    "late-goal",
    "late-exposure"
  ],
  "event_time_order_event_ids": [
    "late-exposure",
    "late-goal"
  ],
  "binary_counts": {
    "control": [
      1,
      1
    ]
  },
  "converter_count": 1,
  "late_delivery_reconciled": true
}

Reproduce: PYTHONPATH=src python3 -m unittest tests.test_integrity_lab.IntegrityLabTests.test_late_row_reconciles_by_event_time_not_receipt_order -v

Reordered

Reordering delivery must not change the canonical event history, attributed counts, or reconciliation digest.

Test: the same assignment, exposure, and goal are delivered in reverse order

Observed pipeline result

{
  "chronological_receipt_order": [
    "reordered-assignment",
    "reordered-exposure",
    "reordered-goal"
  ],
  "reordered_receipt_order": [
    "reordered-goal",
    "reordered-exposure",
    "reordered-assignment"
  ],
  "chronological_counts": {
    "control": [
      1,
      1
    ]
  },
  "reordered_counts": {
    "control": [
      1,
      1
    ]
  },
  "chronological_digest": "a3bd6966fe07838fbe3ea91ed220f5269d70aef772276ff05ec8610b011f9829",
  "reordered_digest": "a3bd6966fe07838fbe3ea91ed220f5269d70aef772276ff05ec8610b011f9829",
  "digests_match": true
}

Reproduce: PYTHONPATH=src python3 -m unittest tests.test_integrity_lab.IntegrityLabTests.test_reordered_row_matches_chronological_reconciliation_digest -v

Cross Version

An outcome may only join an exposure from the same experiment version; cross-version contamination must produce zero conversions.

Test: version 1 exposure and version 2 goal share the same subject

Observed pipeline result

{
  "version_1_binary_counts": {
    "control": [
      1,
      0
    ]
  },
  "version_2_binary_counts": {},
  "version_1_converter_count": 0,
  "version_2_converter_count": 0,
  "cross_version_conversion_leaked": false
}

Reproduce: PYTHONPATH=src python3 -m unittest tests.test_integrity_lab.IntegrityLabTests.test_cross_version_row_prevents_outcome_leakage -v

Scope

Deterministic in-memory SQLite evidence. This lab demonstrates local ingestion and reconciliation semantics, not distributed delivery, Postgres throughput, or availability.

Build and review a test

1Define the decision

Test name What do you expect to happen?

Success metric

Safety metric Maximum acceptable regression

Advanced experiment definition

Eligibility rule Control allocation Stopping rule

Baseline rate Target absolute effect

Typed variables (JSON) Combination and sequence rules (JSON)

Use effect: "exclude" to remove a matching combination. Use match_order: "in_order" to validate the predicates as a sequence in declared factor order. Contradictory references or orderings block launch.

2States3 valid · 1 rejected

Status	Typed values	Constraint result
Valid · Control	`{ "starter_type": "template", "guided_tour": true }`	All constraints passed · allocation 50.0%
Valid · Treatment	`{ "starter_type": "template", "guided_tour": false }`	All constraints passed · allocation 25.0%
Valid · Treatment	`{ "starter_type": "blank", "guided_tour": false }`	All constraints passed · allocation 25.0%
Rejected	`{ "starter_type": "blank", "guided_tour": true }`	Blank starts cannot show the template-guided tour.

3Powern=292 per state

Predeclared inputs

Baseline 10.0% · target effect 8.0% · alpha 0.05 · power 80%

Planning result

Required n=292 per state · implied MDE 7.99%

4Launchdraft v1

Hypothesis: A guided, template-first start increases activation without increasing support contacts.

Decision metric: activated_within_7_days

Guardrail: support_contacts_per_user cannot regress by more than 0.02

Eligibility: new_accounts AND locale IN supported_locales

Allocation: 50% control / 50% treatment

Stopping rule: Fixed horizon: 14 days and required sample size reached; no repeated peeking.

5Data health

Configuration current
Versions agree
Exposure complete
Ingestion delay 2m
Guardrails healthy

6Results

Live SDK telemetry

Launch a version to accept assignment, exposure, and outcome events.

Decision fixture

Launch a reviewed version before viewing effects.

7Decision log

Version	Decision	Created	Frozen evidence
No decisions recorded.

8Lifecycle

Version	Lifecycle	Launched	Immutable configuration
No launched versions yet.