msa-ad

Measurement System Analysis for Automated-Driving Validation.

Gage R&R, attribute agreement, and seeded-fault coverage verification for simulation, hardware-in-the-loop (HIL) and fault-injection benches.

It quantifies the repeatability, reproducibility, reference-relative bias and monitor effectiveness of a bench. It is evidence that supports a tool-validation argument; it does not, on its own, establish that a simulation is accurate or valid for a given operational design domain — that is a separate obligation requiring an independent physical reference. See docs/gap-and-prior-art.md.

Python ≥ 3.10. Depends on numpy, scipy and pandas only. Apache-2.0.

A fixed 60-second slice of the public U.S. DOT/FHWA NGSIM US-101 vehicle-trajectory dataset, replayed and analysed end to end. Runs in seconds:

curl -sL https://github.com/Never2land/msa-ad/archive/refs/heads/main.tar.gz | tar xz \
 && cd msa-ad-main && pip install . -q && python example/ngsim_pilot/run_pilot.py

Or with Docker: docker build -t msa-ad https://github.com/Never2land/msa-ad.git && docker run --rm msa-ad

Expected: %GRR 6.36, ndc 22, seeded faults 100/100 (95% Wilson CI 96.3–100%). What those numbers mean — and what they do not establish — is one page: REPRODUCE.md. This pilot is a reproducible method demonstration on public data, not an industrial bench qualification; the claim boundary is stated in the pilot's own report and in REPRODUCE.md.

Automated-driving safety cases rest on simulation, HIL and fault-injection benches rather than on road mileage — the mileage needed for a statistical demonstration by driving is not drivable inside a development programme (Kalra & Paddock, RAND RR-1478-RC, 2016). That makes the bench the instrument of record, and regulated manufacturing already knows how to treat an instrument of record: before trusting what a gauge says about a part, you characterise the gauge (AIAG MSA), and you pass known-bad master parts through inspection on a schedule to prove the inspection can fail.

msa-ad transfers both practices onto automated-driving validation benches:

1. Bench characterisation — how much of an observed scenario verdict originates in the apparatus rather than in the system under test?

On a bench, "repeat" usually means a new random seed, and seed-driven scatter is aleatory variation of the scenario, not error of the apparatus. The two-tier design replicates once with the seed fixed and once with it varied, and reports the two characterisations side by side rather than as one confounded %GRR.

2. Coverage-claim verification — a catalogue of known faults is injected and run through the production campaign; the fraction actually caught is a measured detection rate reported alongside any coverage claim.

Why this is not already covered by ISO 26262-8 Clause 11 / ISO 21448 — and the standing invitation to prove that claim wrong — is argued in full in docs/gap-and-prior-art.md.

import pandas as pd
from msa_ad import (
 gage_rr_anova, two_tier_gage_rr, attribute_agreement,
 analyse_seeded_faults, compare_claimed_coverage, render_gage_rr_report,
)
runs = pd.read_csv("example/bench_runs.csv")
# Continuous safety-margin metric -> crossed ANOVA Gage R&R
grr = gage_rr_anova(runs, scenario_col="scenario_id",
 bench_col="bench_id", value_col="min_ttc_s")
print(render_gage_rr_report(grr))
# Fixed- vs varied-seed tiers -> instrument vs stochastic model, side by side
two_tier = two_tier_gage_rr(
 pd.read_csv("example/bench_runs_two_tier.csv"),
 value_col="min_ttc_s", tier_col="tier",
 deterministic_benches=["HIL-A", "HIL-B"],
)
# Binary pass/fail verdicts -> attribute agreement, NOT variance decomposition
attr = attribute_agreement(runs, verdict_col="verdict")
# Seeded faults -> measured detection rate with Wilson intervals
seeded = analyse_seeded_faults(
 pd.read_csv("example/fault_catalogue.csv"),
 pd.read_csv("example/campaign_results.csv"),
 threshold=0.80,
)
claim = compare_claimed_coverage(seeded, claimed_coverage=0.95)

Passing binary verdicts to gage_rr_anova raises BinaryDataError rather than returning a meaningless %GRR; passing a single tier to two_tier_gage_rr raises MissingTierError rather than quietly confounding the figure. Both guards are deliberate; the statistics behind them are in docs/statistical-notes.md.

Install for development and run the annotated synthetic example (docs/worked-example.md):

git clone https://github.com/Never2land/msa-ad && cd msa-ad
python -m pip install -e ".[dev]"
python example/run_example.py
python -m pytest # 235 tests, asserted against hand-worked values

Answers to any of these — critical ones especially — are genuinely valuable, as issues here (there is a two-minute reproduction-report template) or by email:

1. Does the crossed design actually isolate bench-contributed variation from vehicle-software variation — or is there a confound you can name?
2. Which statistical assumption here is least credible?
3. What would it take — or what would block you — to run this on your own simulator or replay pipeline? If the harness interface is the blocker, describe the interface and we will look at adapting it.

And the two standing requests from day one: prior art that collapses the gap argument (if a standard or published method already does this, it should be cited), and a correction to the statistics (a failing test is the clearest form).

Please read this section before citing or relying on anything here.

• This is a proposed method and a reference implementation. It is not a standard. It has not been through any standardisation body, industry consortium or formal review.
• It has not been validated against industrial data. I have not run it against a production automated-driving validation programme, and I make no claim about how it behaves on one.
• The bundled example data is entirely synthetic. It is generated from a fixed seed by msa_ad.datagen and was constructed to contain the findings it reports. It is not measured data from any bench, programme or vehicle, and no number in it should be read as an observation about any real system.
• No industrial adoption is claimed. No organisation uses this. No benchmarks are asserted beyond what the code computes on the synthetic data shipped with it.
• The gap argument in this README is a claim, not an established finding. It is my reading of what ISO 26262 and ISO 21448 do and do not prescribe. I may be wrong, and I would like to know if I am.
• Scope limits. The crossed ANOVA requires a balanced design with no empty cells; nested and unbalanced designs are not implemented. Stability monitoring — the control-chart component of AIAG MSA, tracking a bench over time — is described as part of the method but is not implemented here. The bias and linearity study requires reference values that many benches will not have.
• Two-tier caveats. The subtraction assumes the apparatus term is the same in both tiers — that the tiers differ only in whether the seed moves. A bench whose noise depends on the workload violates that, and nothing here detects it. No confidence interval is reported on the aleatory variance: it is a difference of two mean squares, and an exact interval for that is a Behrens–Fisher-shaped problem I have not attempted; the F-test on the ratio is offered instead. With pool_interaction="auto" the two tiers can reach different pooling decisions, which makes the two repeatability terms slightly different quantities; pooling_matches records it and the report says so, but the subtraction is still performed. The determinism audit is evidence only over the scenarios executed — it cannot establish determinism, only refute it.
• Statistical caveats. The interaction pooling rule at alpha = 0.25 is AIAG's convention, not a derived optimum. ndc is a rule of thumb, not an inferential statistic. The kappa bands quoted in reports are Landis and Koch's descriptive conventions and are not acceptance criteria. The Wilson intervals on agreement proportions treat scenarios as independent, which repeated scenario families may violate.

• REPRODUCE.md — the one-command pilot, expected values, and a one-page interpretation
• docs/gap-and-prior-art.md — the gap argument in full: what EU/UNECE instruments leave open, why this is not ISO 26262-8 Clause 11, where this sits in UL 4600, references
• docs/worked-example.md — annotated output of the synthetic end-to-end example (constructed to contain findings)
• docs/statistical-notes.md — the variance decomposition, two-tier subtraction, determinism audit, kappa and Wilson choices, and how the 235-test suite asserts them against hand-worked values

Machine-readable metadata in CITATION.cff. An archived, citable snapshot is on Zenodo — cite the concept DOI, which resolves to the latest archived version:

Wang, L. msa-ad: Measurement System Analysis for Automated-Driving Validation. Zenodo. DOI: 10.5281/zenodo.21963049

To pin the exact code you ran, cite that release's version DOI instead.

Linlin Wang. Licensed under the Apache License, Version 2.0. See LICENSE.