Robustness testing with ISO/IEC TR 24029-1

This guide runs the statistical and empirical robustness assessment of a high-risk AI system under Art. 15(4) with the method categories of ISO/IEC TR 24029-1:2021Published, following the steps of PR-15-ROB v1.0 that cover the environment manifest, the perturbation sweep and the confidence interval. The guide gives the goal statement, the sweep, the interval estimate and the comparison with the declared level. At the end the reader holds one robustness test report entry per goal, with the sample size, the interval and the record that makes the run reproducible.

What the obligation asks for

Art. 15(4), first subparagraph, of Regulation (EU) 2024/1689 requires a high-risk AI system to be as resilient as possible regarding errors, faults or inconsistencies that may occur within the system or in its operating environment, in particular due to its interaction with natural persons or other systems. The Art. 15 obligation record lists the testable element this guide covers: outputs stay within the declared performance under input perturbation, noise, malformed input, edge cases and inconsistent signals from natural persons or connected systems. Two neighbouring elements use the same machinery. The consistency element under Art. 15(1) runs the same sweep on a distribution-shifted set and on each released model version, and the declaration under Art. 15(3) supplies the level the sweep result is compared with. Regulation (EU) 2026/1744 did not change the content of Art. 15; it moved the Annex III application date 2027-12-02Deferred and the Annex I application date 2028-08-02Deferred that a manufacturer of an Annex I product follows.

The Act names no method. The method categories come from ISO/IEC TR 24029-1:2021Published, a published technical report listed on the standards page, which groups robustness assessment into statistical, formal and empirical methods and gives the form of a robustness goal: a property, an input region, a metric and a bound. A statistical method samples perturbed inputs and estimates the metric with a confidence interval; an empirical method runs the system under a recorded condition and observes the output; a formal method proves a property over a bounded input region. This guide covers the first two. The formal route follows ISO/IEC 24029-2:2023Published, a published international standard, in step 19 of the protocol and in the separate guide on formal robustness verification with that standard. The perturbation classes a sweep covers come from prEN 18229-2Draft, a JTC 21 draft that is evidence of the state of the art and binds nothing. No harmonised standard is cited in the Official Journal for Art. 15 as of the anchor date, so no presumption of conformity attaches to any method in this guide.

Inputs

The protocol page for PR-15-ROB v1.0 lists every input. The statistical and empirical route needs the following from that list.

Model access at a fixed version: an inference endpoint or container image with the version hash and the interface specification.
Instructions for use with the Art. 13(3)(b)(ii) declaration: metric names, levels, test conditions and known circumstances.
Training, validation and test data with a dataset card per set: source, collection period, labelling method, split rule.
The condition register from the risk management file: operating conditions, foreseeable misuse conditions and known circumstances.
The validation and testing procedures description from Annex IV section 2(g), where one exists, so that the sweep reuses the declared metric definitions.
Write access to a run directory for the manifest, the provenance record, the run logs and the report.

Procedure

Steps 1 to 12 below are the statistical and empirical subset of PR-15-ROB, renumbered for this guide; the cite line on each step names the protocol step and the clause it derives from. Every path, hash, model, endpoint and number in the code blocks is a synthetic example, and none is a requirement of the Act.

Record the model version hash, inference configuration, random seeds and tool versions in one environment manifest.
Record Environment manifest with the model version hash
[PR-15-ROB step 1 (AIAssuranceLabs procedure)]

Example for step 1, a synthetic manifest built with sha256sum, pip and jq; the model file, seed and paths are placeholders.

# synthetic example: environment manifest for a tabular classifier served from an ONNX file
mkdir -p manifest
sha256sum models/credit-scorer-v3.onnx | cut -d' ' -f1 > manifest/weights.sha256
python -c "import sys, numpy, onnxruntime; print(sys.version.split()[0], numpy.__version__, onnxruntime.__version__)" > manifest/tool-versions.txt
pip freeze > manifest/requirements.lock
jq -n \
  --arg system "credit-scorer" \
  --arg version "v3" \
  --arg weights "$(cat manifest/weights.sha256)" \
  --arg tools "$(cat manifest/tool-versions.txt)" \
  --arg lock "$(sha256sum manifest/requirements.lock | cut -d' ' -f1)" \
  '{system: $system, model_version: $version, weights_sha256: $weights,
    inference: {batch_size: 64, threads: 4, threshold: 0.5},
    seeds: {numpy: 4711}, tool_versions: $tools, requirements_lock_sha256: $lock}' \
  > manifest/environment.json

Extract the declared accuracy metrics, levels and test conditions from the instructions for use.
Record Declared-level extract with metric names, levels and test conditions
[Regulation (EU) 2024/1689, Art. 15(3) and Art. 13(3)(b)(ii); PR-15-ROB step 3]
State each robustness goal as a property, an input region, a metric and a bound.
Record Robustness goal statement per goal
[ISO/IEC TR 24029-1:2021, clause on robustness goals and their statement; PR-15-ROB step 6]

Example for step 3, one synthetic goal statement for a tabular credit-scoring classifier; the magnitudes, the bound and the sample count are the provider's inputs to this example, recorded with their source in the target justification record of the protocol.

# synthetic example: robustness goal RG-03
goal_id: RG-03
property: predicted class is unchanged under additive noise on the six income and balance features
input_region:
  base_set: heldout-v3            # hash in the test set provenance record
  perturbation_class: noise       # prEN 18229-2 draft class; see the perturbation class register
  distribution: gaussian, zero mean
  magnitudes_sigma: [0.01, 0.02, 0.05, 0.10]   # fraction of each feature's training-set standard deviation
metric: agreement rate between the perturbed prediction and the unperturbed prediction
bound: lower limit of the two-sided interval at or above 0.97 at every magnitude
confidence: 0.95
samples_per_magnitude: 2000
method: statistical                # ISO/IEC TR 24029-1:2021 category; reason in the method selection record
bound_source: provider target, source hierarchy level 6, justification in the target justification record

Select a statistical or empirical method for each goal and record the reason.
Record Method selection record with the reason per goal
[ISO/IEC TR 24029-1:2021, clauses on statistical and empirical methods; PR-15-ROB step 7]
Assemble a held-out set untouched by training or model selection; record sample counts and hashes.
Record Test set provenance record with sample counts and hashes
[PR-15-ROB step 10 (AIAssuranceLabs procedure)]
Define each perturbation class and its magnitudes: noise, malformed input, edge cases and inconsistent signals.
Record Perturbation class register with the magnitudes per class
[prEN 18229-2, draft, clause on perturbation classes for robustness assessment; PR-15-ROB step 17]
Run the sweep: sample perturbed inputs at each magnitude and record every base and perturbed output.
Record Perturbation run log per class and magnitude
[PR-15-ROB step 17]

Example for step 7, a synthetic sweep for goal RG-03 against a local inference endpoint; the files, the endpoint and the seed match the manifest above.

# synthetic example: perturbation sweep for goal RG-03 against a local endpoint
import csv
import numpy as np
import requests

rng = np.random.default_rng(4711)                 # seed from manifest/environment.json
X = np.load("data/heldout-v3.npy")                # held-out features, provenance record HO-03
scale = np.load("data/train-feature-std.npy")     # per-feature standard deviation of the training set
URL = "http://localhost:8080/predict"

def predict(batch):
    r = requests.post(URL, json={"instances": batch.tolist()}, timeout=30)
    r.raise_for_status()
    return np.array(r.json()["labels"])

base = predict(X)
with open("runs/RG-03-sweep.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["sigma", "sample", "index", "base", "perturbed"])
    for sigma in [0.01, 0.02, 0.05, 0.10]:
        idx = rng.choice(len(X), size=2000, replace=True)
        Xp = X[idx] + rng.normal(0.0, sigma, X[idx].shape) * scale
        out = predict(Xp)
        for s, i in enumerate(idx):
            w.writerow([sigma, s, int(i), int(base[i]), int(out[s])])

Estimate the goal metric at each magnitude with a confidence interval and the sample size.
Record Statistical robustness estimate with interval and sample size
[ISO/IEC TR 24029-1:2021, clause on statistical methods; PR-15-ROB step 18]

Example for step 8, the agreement rate per magnitude with a Wilson score interval computed from the sweep log; the confidence level is the one recorded in goal RG-03.

# synthetic example: agreement rate per magnitude with a Wilson interval, from runs/RG-03-sweep.csv
import csv
import math
from collections import defaultdict

Z = 1.959964            # two-sided normal quantile for the 0.95 confidence recorded in goal RG-03
BOUND = 0.97            # lower-limit bound recorded in goal RG-03
hits, n = defaultdict(int), defaultdict(int)
with open("runs/RG-03-sweep.csv", newline="") as f:
    for row in csv.DictReader(f):
        n[row["sigma"]] += 1
        hits[row["sigma"]] += int(row["base"] == row["perturbed"])

def wilson(k, m, z=Z):
    p = k / m
    d = 1 + z * z / m
    centre = (p + z * z / (2 * m)) / d
    half = z * math.sqrt(p * (1 - p) / m + z * z / (4 * m * m)) / d
    return centre - half, centre + half

for sigma in sorted(n, key=float):
    lo, hi = wilson(hits[sigma], n[sigma])
    print(f"sigma={sigma} n={n[sigma]} rate={hits[sigma] / n[sigma]:.4f} "
          f"ci=[{lo:.4f}, {hi:.4f}] meets={'yes' if lo >= BOUND else 'no'}")

The synthetic output below is what the script prints for an invented sweep log; the values carry no measured result.

sigma=0.01 n=2000 rate=0.9955 ci=[0.9916, 0.9976] meets=yes
sigma=0.02 n=2000 rate=0.9890 ci=[0.9836, 0.9927] meets=yes
sigma=0.05 n=2000 rate=0.9745 ci=[0.9668, 0.9805] meets=no
sigma=0.10 n=2000 rate=0.9310 ci=[0.9190, 0.9414] meets=no

For an empirical goal, run the system under the recorded condition and record the output per case.
Record Condition run log with a result per case
[ISO/IEC TR 24029-1:2021, clause on empirical methods; PR-15-ROB step 16]
Compare each estimate with the bound in its goal and the declared level; record meets or does not meet.
Record Declared-level comparison table
[Regulation (EU) 2024/1689, Art. 15(3); PR-15-ROB step 33]
Write the robustness report with statistical and empirical results grouped by method category.
Record Robustness test report
[ISO/IEC TR 24029-1:2021, clause on assessment methods; PR-15-ROB step 40]
Record the model, data and environment changes that invalidate the report; archive every record with its hash.
Record Lifecycle performance monitoring record with re-test triggers
[Regulation (EU) 2024/1689, Art. 15(1); PR-15-ROB step 45]

A pitfall that vendor documentation omits

A perturbation sweep reports a point estimate per magnitude, and evaluation tooling stops there. A statistical method in the sense of ISO/IEC TR 24029-1:2021Published, the published technical report, produces an estimate from a sample, and the estimate carries the uncertainty of that sample. The Limitations section of PR-15-ROB v1.0 records the confidence interval and the sample size with every statistical estimate and states that the estimate is not a proof. The consequence for step 10 shows in the synthetic output above: at the third magnitude the point estimate sits above the bound and the lower limit of the interval sits below it, so the goal is recorded as not met at the confidence the goal states. The goal statement in step 3 names which limit the comparison uses, and the comparison table records the interval and the sample size next to the point value, so that an assessor reads both before the result.

The bound itself has no source in the Act. Art. 15(1) requires an appropriate level and Art. 15(3) requires the provider to declare it; the obligation record's threshold note states that neither the Act nor any document in the source hierarchy publishes a numeric pass level for a high-risk system. The hierarchy runs: a harmonised standard cited in the Official Journal, of which none is cited as of the anchor date; a JTC 21 draft such as prEN 18229-2Draft for the metric categories and perturbation classes; an ISO/IEC document such as ISO/IEC TR 24029-1:2021Published for the method categories; a Commission guideline or code of practice; NIST AI RMF 1.0Published for the metric selection rationale; and the state of the art the provider justifies. The target justification record in the protocol holds the level chosen and the source level it came from, and a bound in a goal statement without that record is a number without a condition.

What the result shows and what it does not

A result of this procedure holds for the model version in the environment manifest, the sets in the test set provenance record and the perturbation classes and magnitudes in the register; a change to any of these invalidates the result, and step 12 records the triggers. A statistical estimate holds at the recorded confidence for the sampled region and says nothing about inputs outside the magnitudes swept. An empirical result holds for the conditions run. The procedure measures against the declared level and the bound the provider set; it cannot show that the level or the bound is the appropriate one for the intended purpose, a judgement that belongs to the provider under Art. 15(1) and to its notified body and counsel where a conformity assessment follows. The report is evidence for the provider's own technical file under Annex IV section 2(g), and it is the test report delivered by the robustness and adversarial testing service within the limits stated on what we cannot do.

The obligation, the protocol and the ICP hubs for this guide are listed below; the methodology index holds every protocol with its version.

Obligations: Art. 15
Protocols: PR-15-ROB v1.0
Related guides: Formal robustness verification with ISO/IEC 24029-2:2023
ICP hubs: Annex III providers
Annex I manufacturers

Questions

What is ISO/IEC 24029?

ISO/IEC 24029 is the ISO/IEC JTC 1/SC 42 series on the assessment of the robustness of neural networks. Part 1, ISO/IEC TR 24029-1:2021, is a published technical report that gives an overview of assessment methods in three categories: statistical, formal and empirical. Part 2, ISO/IEC 24029-2:2023, is a published international standard on the use of formal methods. No harmonised standard is cited in the Official Journal for Art. 15 as of the anchor date, so no presumption of conformity attaches to Part 1 or Part 2; both parts serve as method references for an Art. 15 robustness test.

What does ISO/IEC TR 24029 cover?

ISO/IEC TR 24029-1:2021, the published technical report, covers methods for assessing whether a neural network keeps its intended level of performance when its inputs move away from the typical inputs it was trained on. The report names the method categories, the form of a robustness goal and the metrics a statistical assessment reports. The report sets no pass level and requires no method. In PR-15-ROB the report supplies the goal statement in step 6, the method selection in step 7 and the interval estimate in step 18.

Every inquiry receives a reply within 24 hours. Scope and availability are discussed per request.

Ask about scope hello@aiassurancelabs.com