AI bias testing for fairness under the EU AI Act: subgroup disaggregation

What the obligation asks for

Art. 10(2)(f) of Regulation (EU) 2024/1689 requires the data governance practices for a high-risk AI system to cover an examination in view of possible biases that are likely to affect the health and safety of persons, to have a negative impact on fundamental rights, or to lead to discrimination prohibited under Union law, especially where data outputs influence the inputs for future operations. Art. 10(2)(g) requires appropriate measures to detect, prevent and mitigate those biases. The obligation record lists two testable elements that this guide covers: the bias examination result, which is system performance and outcomes disaggregated by protected and context-relevant subgroup with each metric defined, and the effect of the mitigation measures, verified by re-running the disaggregated evaluation. The subgroup counts from the representativeness check in Art. 10(3) are the denominators of that evaluation. The duties apply from the Annex III application date 2027-12-02Deferred. Art. 10 sets no fairness metric, no minimum sample size per subgroup and no acceptable disparity between groups.

Disaggregation needs a protected attribute per record. Where that attribute is a special category of personal data, Art. 4a of Regulation (EU) 2024/1689, the provision that Regulation (EU) 2026/1744 moved from the former Art. 10(5), states the legal basis and the conditions in points (a) to (f). The necessity threshold in Art. 4a is an unsettled point, and this guide takes no reading of its own; it shows both readings as recorded and refers the question to the provider and its counsel.

Inputs

The list follows the Inputs section of the protocol page.

• The testing data set, one file with a schema, labels and a record identifier, and its file hash.
• Model access for inference at a fixed version, or the output log of that version keyed by record identifier.
• The protected attribute fields and the context-relevant subgroup fields, with a coding table per attribute.
• The intended purpose and the instructions for use, with the persons or groups the system is intended to be used for.
• The provider's disparity target per metric and the minimum cell size, each with its justification.
• Where special categories of personal data are present, the records of processing activities and the safeguards description.
• A Python environment with pandas, fairlearn as an example of a disaggregation library, and statsmodels for the intervals.

Procedure

The steps below are the subset of PR-10-BIAS that produces the disaggregation tables. The step numbers of the protocol appear in each citation.

1. Define with the provider the protected attributes and the context-relevant subgroups for disaggregation.

   Record Subgroup definition table with the coding of each attribute value and the reason for each attribute

   [ISO/IEC TR 24027:2021 §5.3, fairness (PR-10-BIAS step 7)]

2. Select the fairness metrics for the context of use and write the definition of each metric.

   Record Metric selection record with one definition per metric

   [ISO/IEC TR 24027:2021 §7.2, fairness metrics (PR-10-BIAS step 9)]

3. Record the provider's disparity target and minimum cell size with the justification for each.

   Record Threshold justification record with the disparity target, the minimum cell size and the source level of each

   [AIAssuranceLabs procedure]

The metric choice in step 2 is the provider's, and the record states why. ISO/IEC TR 24027:2021Published lists candidate metrics built on the confusion matrix and states no pass level for any of them. The table gives four of those candidates with what each one shows.

The target for each selected metric comes from the source hierarchy in the Art. 10 obligation record. No harmonised standard for Art. 10 is cited in the Official Journal as of the anchor date, so the first level is empty. prEN 18283Draft supplies bias management concepts and candidate measures at draft stage, with no public text and no pass level. ISO/IEC TR 24027:2021Published supplies the metric definitions at the third level and states no acceptable disparity. NIST AI RMF 1.0Published supplies, in its Measure function, the expectation that the metric selection rationale is documented. Below those levels sits the state of the art that the provider justifies against the context of use, which is what step 3 records. Several of the metrics in the table cannot be equal across subgroups at the same time when the base rates differ between the subgroups, so the metric selection record states which metrics were selected and which were set aside, with the reason.

4. Hash the testing data set and record the model version identifier.

   Record File hash of the testing data set and the model version identifier (PR-10-BIAS steps 3 and 22)

   [AIAssuranceLabs procedure]

5. Run the model at the recorded version on every record of the testing data set.

   Record Output log with the model version (PR-10-BIAS step 22)

   [AIAssuranceLabs procedure]

6. Compute the confusion matrix per subgroup for each protected attribute.

   Record Disaggregated confusion matrices, one per subgroup and attribute

   [ISO/IEC TR 24027:2021 §7.2, confusion matrix (PR-10-BIAS step 23)]

7. Compute each selected metric per subgroup and the disparity between each subgroup and the reference subgroup.

   Record Fairness metric and disparity table

   [ISO/IEC TR 24027:2021 §7.2, fairness metrics (PR-10-BIAS step 24)]

8. Compute each selected metric for each intersectional cell and flag each cell below the minimum cell size.

   Record Intersectional metric table with a sample-size flag per cell

   [prEN 18283, draft, clause on bias measures (PR-10-BIAS step 25)]

9. Compute a confidence interval for each cell metric and store the interval next to the point value.

   Record Interval columns in the intersectional metric table

   [AIAssuranceLabs procedure]

10. Compare each disparity with its justified target and mark each unflagged result pass or fail.

    Record Disparity findings, pass or fail per metric and subgroup pair (PR-10-BIAS step 33)

    [AIAssuranceLabs procedure]

11. Write the protocol version, the file hash and the model version into the header of every table.

    Record Report provenance block (PR-10-BIAS step 44)

    [AIAssuranceLabs procedure]

Example for step 4. The paths, the endpoint and the file are synthetic; the model runs on a local endpoint that returns its version identifier.

# synthetic example: fix the inputs before any metric is computed
mkdir -p provenance out
sha256sum data/test_set.csv > provenance/test_set.sha256
curl -s http://localhost:8080/version | jq -r '.model_version' > provenance/model_version.txt

The scored file for the examples below has one row per record with the label, the prediction and the two protected attributes. The rows are synthetic.

record_id,sex,age_band,y_true,y_pred
r0001,female,18-34,1,1
r0002,male,35-49,0,0
r0003,female,50+,1,0
r0004,male,50+,1,1
r0005,female,35-49,0,1

Example for steps 6 to 8. The script computes the cell size, the number of positive records, the true positive rate, the false positive rate and the selection rate, first per attribute and then per intersection of the two attributes. The minimum cell size in the script is an input of the example taken from the threshold justification record of step 3; the Act states no such number.

# synthetic example: disaggregation with pandas and fairlearn MetricFrame
import numpy as np
import pandas as pd
from fairlearn.metrics import (
    MetricFrame,
    count,
    false_positive_rate,
    selection_rate,
    true_positive_rate,
)

df = pd.read_csv("data/test_scored.csv")

def positives(y_true, y_pred):
    return int(np.asarray(y_true).sum())

metrics = {
    "n": count,
    "positives": positives,
    "tpr": true_positive_rate,
    "fpr": false_positive_rate,
    "selection_rate": selection_rate,
}

# step 7: one protected attribute at a time, with the between-group disparity
by_sex = MetricFrame(
    metrics=metrics,
    y_true=df["y_true"],
    y_pred=df["y_pred"],
    sensitive_features=df["sex"],
)
print(by_sex.by_group)
print(by_sex.difference(method="between_groups"))

# step 8: the intersection of two attributes; by_group carries a MultiIndex
inter = MetricFrame(
    metrics=metrics,
    y_true=df["y_true"],
    y_pred=df["y_pred"],
    sensitive_features=df[["sex", "age_band"]],
)
table = inter.by_group.reset_index()

MIN_CELL = 50  # from the threshold justification record, step 3; not from the Act
table["below_min"] = table["n"] < MIN_CELL
table.to_csv("out/intersectional_metrics.csv", index=False)

Example for step 9. The interval is a Wilson score interval on the true positive rate, computed on the number of positive records in the cell, which is the denominator of that metric. The confidence level is an input of the example.

# synthetic example: a Wilson score interval per cell on the true positive rate
from statsmodels.stats.proportion import proportion_confint

def tpr_interval(row):
    n_pos = int(row["positives"])
    if n_pos == 0:
        return (float("nan"), float("nan"))
    k = round(row["tpr"] * n_pos)
    return proportion_confint(k, n_pos, alpha=0.05, method="wilson")

table[["tpr_low", "tpr_high"]] = table.apply(tpr_interval, axis=1, result_type="expand")
table.to_csv("out/intersectional_metrics.csv", index=False)

An excerpt of the resulting table for the synthetic data set. The values are outputs of the example and carry no meaning outside it.

sex,age_band,n,positives,tpr,tpr_low,tpr_high,below_min
male,35-49,540,412,0.825,0.786,0.859,False
female,35-49,470,355,0.839,0.798,0.874,False
male,50+,70,41,0.756,0.607,0.862,False
female,50+,62,38,0.684,0.525,0.809,False
female,18-34,15,12,0.750,0.468,0.911,True

A pitfall that vendor documentation omits

The library documentation describes how to compute a metric per group; it does not say how many records a group needs before the value means anything, and neither does the Act. Art. 10 states no minimum sample size per subgroup, so the minimum cell size is the provider's justified choice, recorded in step 3 with its source level. The pitfall is where that minimum is applied. In the excerpt above, the cell for women aged fifty and over has 62 records and passes a minimum of 50 on the cell count, yet its true positive rate rests on 38 positive records and its Wilson interval spans 0.525 to 0.809. A disparity target of 0.05 absolute against the reference cell would mark the point value of 0.684 as a fail, while the interval also contains values that would pass. For a rate-based metric the relevant count is the denominator of the metric, which for the true positive rate is the number of positive records and for the false positive rate the number of negative records. The threshold justification record therefore states the minimum per metric denominator, and step 8 flags on that denominator.

The limitation is stated in the Limitations section of PR-10-BIAS v1.0: subgroup metrics on small cells carry wide uncertainty, step 25 flags each cell below the recorded minimum sample size, and the protocol does not report a value for a flagged cell as a finding. Storing the interval next to the point value, as step 9 does, makes the width visible to the assessor who reads the table and prevents a narrow disparity target from being applied to a wide interval. Where a cell is flagged, the report records the gap for the data gaps and shortcomings register under Art. 10(2)(h), and the mitigation recommendation names the additional records needed for the cell.

What the result shows and what it does not

The table shows disparity on the subgroups defined in step 1 and the metrics selected in step 2; it does not show the absence of bias on subgroups the data set does not identify or on metrics that were not selected. Several fairness metrics cannot be met at the same time on the same data set, and the metric selection record states which were selected and why; the procedure does not resolve the conflict between them. Where the provider holds no protected attribute data, the disaggregation cannot run on that attribute, and the gap is recorded for the Art. 10(2)(h) register without inferring the attribute. A pass against the justified disparity target is a result against that target only. The table is evidence for the technical documentation under Art. 11 and Annex IV, specified on the deliverables page as the bias examination report with subgroup disaggregation tables; the decision whether the system meets Art. 10 remains with the provider, its notified body where Art. 43 requires one, and its counsel. The bias and fairness testing service runs the full protocol, within the limits stated on what we cannot do.

Related pages

The obligation, the protocol and the ICP hubs this guide serves are listed below, and the methodology index lists every protocol with its version.

Obligations

• Art. 10

Protocols

• PR-10-BIAS v1.0

Related guides

• EU AI Act Article 12 logging: verifying log completeness

ICP hubs

• Annex III providers
• Deployers

Every inquiry receives a reply within 24 hours. Scope and availability are discussed per request.

Ask about scopehello@aiassurancelabs.com