Bias and fairness testing under the Omnibus sensitive-data provision
We test high-risk AI systems against the bias examination in Art. 10(2)(f) and the mitigation measures in Art. 10(2)(g) of Regulation (EU) 2024/1689, and we verify the special-category data safeguards in Art. 4a, the provision Regulation (EU) 2026/1744 moved from the former Art. 10(5). The procedure is PR-10-BIAS v1.0. The client receives test reports with subgroup disaggregation tables and fairness metrics, a findings register with re-test criteria, and an assessment memo on the legal basis and safeguards. The duties apply from the Annex III application date Deferred.
Deliverables
The set is designed to answer the questions an assessor asks under Art. 10(2)(f), Art. 10(2)(g) and Art. 4a(1)(f): which subgroups were examined, with which metrics and which targets, what the disparities were, what was done about them, and on what legal basis special-category data was processed. Each item is specified on the deliverables page as a test report, a findings register or an assessment memo.
| Deliverable | Type | Format | Designed to satisfy |
|---|---|---|---|
| Subgroup performance disaggregation tables by protected attribute | Test report | PDF with Markdown source; CSV result tables; JSON run manifest | |
| Disparate-impact and fairness-metric report with metric definitions and chosen thresholds justified against context of use | Test report | PDF with Markdown source; CSV result tables; JSON run manifest | |
| Documentation of the legal basis and safeguards applied when special-category data was processed for bias detection | Assessment memo | PDF with Markdown source | |
| Mitigation recommendations with re-test criteria | Findings register | XLSX; CSV; PDF summary |
Who this is for
- Annex III providers
- Deployers
A provider of an Annex III system such as recruitment screening or credit scoring carries the data governance duties of Art. 10, and the Act gives that provider no fairness metric, no minimum subgroup size and no acceptable disparity; the provider sets each target and an assessor checks the justification and the evidence behind it. A deployer of such a system holds the input data and the outcome records that show how the system treats the people it is used on, and Art. 4a(2) opens the special-category data basis to deployers under the same conditions as providers, without creating a duty to run bias detection. The Art. 10 duties follow the Annex III application date, and the Art. 4a legal basis follows the Omnibus entry into force; both entries are rendered below with their dates.
- DeferredAnnex III standalone high-risk obligations
- Working readingArticle 4a (formerly Article 10(5)) legal basis for processing special-category data for bias detection and correction, necessity threshold
Scope
The engagement covers the training, validation and testing data sets of one high-risk AI system at a fixed model version, and the outputs of that version on the testing data set, disaggregated by the protected attributes and the context-relevant subgroups the provider selects. Where the system does not train a model, Art. 10(6) applies the criteria to the testing data sets only, and the engagement runs on those sets. Where special categories of personal data are present in a data set or a pipeline stage, the engagement observes the safeguards in Art. 4a(1)(b) to (e) in the pipeline and reviews the legal-basis record against Art. 4a(1)(a) and (f). The live input profile a deployer keeps under Art. 26(4) belongs to data governance and dataset quality testing and is out of scope here.
The metrics and the bias taxonomy come from three documents, listed with their stage on the standards page. ISO/IEC TR 24027:2021Published supplies the bias source categories and the candidate fairness metrics. prEN 18283Draft supplies the bias management vocabulary and candidate measures; it is a JTC 21 draft with no public text. prEN 18284Draft supplies the dataset quality characteristics behind the representativeness, error and completeness checks; it is also a JTC 21 draft. No harmonised standard for Art. 10 is cited in the Official Journal, so a result against any of these documents is evidence for the technical documentation and carries no presumption of conformity.
The engagement is testing. It does not decide whether the processing of special-category data is lawful under Regulation (EU) 2016/679; it records what the pipeline shows and what the records of processing activities state, for the provider and its counsel. It does not select the protected attributes; it records the provider's selection and the reason for each attribute. It is not a conformity assessment and no step in it replaces the procedure under Art. 43; the limits are stated on what we cannot do.
The necessity threshold in Art. 4a is an unsettled point. Step 37 of PR-10-BIAS v1.0 checks the legal-basis record against the working reading below, and the assessment memo states which reading the record meets.
What we test
The table lists each testable element of Art. 10 that this service covers, the protocol that tests it and the artifacts it produces for the record.
| Obligation | Testable element | Protocol | Artifacts |
|---|---|---|---|
| Art. 10 | Representativeness of the training, validation and testing data sets against the persons or groups of persons the system is intended to be used for, measured as subgroup coverage against the declared intended purpose (Art. 10(3)) | PR-10-BIAS v1.0 |
|
| Error rate and completeness of the data sets, measured by label-error sampling and missing-value profiling against the targets the provider declares (Art. 10(3)) | PR-10-BIAS v1.0 | ||
| Statistical properties of the data sets, compared with the documented assumptions about what the data measure and represent (Art. 10(2)(d) and 10(3)) | PR-10-BIAS v1.0 | ||
| Coverage of the geographical, contextual, behavioural or functional setting of intended use in the data sets, checked against the setting declared in the intended purpose (Art. 10(4)) | PR-10-BIAS v1.0 | ||
| Bias examination result: system performance and outcomes disaggregated by protected and context-relevant subgroup, with each metric defined (Art. 10(2)(f)) | PR-10-BIAS v1.0 | ||
| Effect of the bias mitigation measures, verified by re-running the disaggregated evaluation after mitigation (Art. 10(2)(g)) | PR-10-BIAS v1.0 | ||
| Feedback-loop check where data outputs influence inputs for future operations, comparing subgroup distributions across operation cycles (Art. 10(2)(f)) | PR-10-BIAS v1.0 | ||
| Special-category data handling check: pseudonymisation and re-use limits present, access restricted to authorised persons, no transfer to other parties, observed in the data pipeline (Art. 4a(1)(b), (c) and (d)) | PR-10-BIAS v1.0 | ||
| Art. 26 | The deployed configuration, the input types and the use cases fall within the intended purpose and the conditions of use stated in the provider's instructions for use (Art. 26(1)) | PR-10-BIAS v1.0 |
|
| The human oversight measures indicated by the provider under Art. 14(3)(b) are implemented in the deployment and can be exercised: stop, override, disregard the output, and intervene on the operation (Art. 26(2) and 26(3)) | PR-10-BIAS v1.0 | ||
| The natural persons assigned to human oversight hold the authority to act: a stop or override instruction from an assigned person takes effect on the system in a test (Art. 26(2)) | PR-10-BIAS v1.0 | ||
| The assigned persons have the necessary support in the deployed workflow: access to the instructions for use, the output information the provider declares under Art. 13, and an escalation path, verified in a walkthrough (Art. 26(2)) | PR-10-BIAS v1.0 | ||
| Where the deployer controls the input data, the live input population and feature distributions are relevant and sufficiently representative for the intended purpose declared in the instructions for use (Art. 26(4)) | PR-10-BIAS v1.0 | ||
| The suspension of use in Art. 26(5) can be executed: a suspension instruction halts the use of the system in the deployment in a test (Art. 26(5)) | PR-10-BIAS v1.0 | ||
| The logs automatically generated by the system and under the deployer's control are complete and readable: a sampled decision can be reconstructed from the retained logs (Art. 26(6)) | PR-10-BIAS v1.0 | ||
| The retention configuration of the log store enforces the period of at least six months in Art. 26(6), or the different period that applicable Union or national law sets (Art. 26(6)) | PR-10-BIAS v1.0 |
Art. 10 sets no numeric threshold, so every pass criterion comes from a source hierarchy and the record for each criterion names its level. The first level is a harmonised standard cited in the Official Journal; none exists for Art. 10 as of the anchor date. The second level is a JTC 21 draft, stated with its stage. The third is an ISO/IEC standard or technical report. The fourth is a Commission guideline or code of practice, with its publication date. The fifth is NIST AI RMF or a sector framework. The sixth is the state of the art the provider justifies and documents. For coverage, error rate, completeness and disparity, no published threshold exists at any level, so step 10 of the protocol records the provider's target with its justification and its source level, and steps 32 and 33 compare each measured value with that target.
Protocols used
One protocol covers this service, and its page states the derivation of every step from the documents named above.
- PR-10-BIAS v1.0Bias and subgroup performance protocolv1.0
PR-10-BIAS v1.0 profiles each data set for lineage, subgroup coverage, setting coverage, label error rate, completeness and statistical properties, runs the model on the testing data set, computes the confusion matrix and each selected fairness metric per subgroup and per intersectional cell, and observes the Art. 4a(1) safeguards in the pipeline. It produces the dataset quality profile, the bias examination report with disaggregation tables, the mitigation measures record with post-mitigation deltas, the legal-basis record and the safeguards and deletion record. It shows disparity only on the subgroups and metrics selected at steps 7 and 9; it does not show the absence of bias on attributes the data sets do not identify, it does not resolve the conflict between fairness metrics that cannot be met at the same time, and it reports no value as a finding for a cell below the recorded minimum sample size.
Engagement shape
Fixed-scope, scoped per request. The classification and readiness assessment is the entry service and precedes this one where the obligation set of the system has not been mapped. The client provides the data sets in scope with a schema and a record identifier, the intended purpose and instructions for use, the dataset documentation under Art. 10(2)(a) to (e), model access at a fixed version, the protected attribute fields with a coding table, the coverage, error, completeness and disparity targets with a justification for each, and, where special-category data is present, the records of processing activities and the safeguards description.
The work follows the order of the protocol. Preparation records the scope, the training-technique statement under Art. 10(1) and 10(6), the subgroup definitions, the bias source register, the metric selection and the threshold justification. Execution traces lineage, counts coverage, samples label errors, profiles completeness, runs the model, computes the disaggregated metrics and observes the Art. 4a safeguards. Analysis compares each value with its target, attributes each failed finding to a bias source, records the mitigation measures and re-runs the disaggregated evaluation. Reporting assembles the deliverables above with a provenance block that carries the protocol version, the file hashes and the model version.
What we cannot do
- We do not perform conformity assessments where a notified body is required.
- We do not issue CE marking or the EU Declaration of Conformity.
- We are not a notified body.
- We do not give legal opinions on classification or liability.
- We test against requirements, prepare evidence, and structure documentation; compliance decisions remain with the provider or deployer, their notified body, and their counsel.
For this service, a pass against a justified disparity target is a result against that target only; the decision whether the system meets Art. 10 remains with the provider, its notified body where Art. 43 requires one, and its counsel. The assessment memo on the legal basis records what the pipeline and the records show against Art. 4a(1); it is not a legal opinion on lawfulness under Regulation (EU) 2016/679.
What the client must have ready
The elements below are documentation and process elements of Art. 10 that the protocol records as inputs and does not test.
Art. 10Data and data governance
- Statement of whether the system uses techniques involving the training of AI models, which determines whether the criteria apply to all data sets or to the testing data sets only (Art. 10(1) and 10(6)) documentation
- Design choices record for the data sets (Art. 10(2)(a)) documentation
- Data collection and origin record, including the original purpose of collection for personal data (Art. 10(2)(b)) documentation
- Data-preparation record listing the annotation, labelling, cleaning, updating, enrichment and aggregation operations (Art. 10(2)(c)) documentation
- Statement of assumptions about the information the data are supposed to measure and represent (Art. 10(2)(d)) documentation
- Assessment of the availability, quantity and suitability of the data sets that are needed (Art. 10(2)(e)) documentation
- Record of identified data gaps or shortcomings and how they are addressed (Art. 10(2)(h)) documentation
- Justification of the representativeness, error and completeness targets against the intended purpose, and a statement of whether the characteristics are met per data set or by a combination of data sets (Art. 10(3)) documentation
- Record of the reasons why processing special categories of personal data was strictly necessary and why other data, including synthetic or anonymised data, could not achieve the objective, in the records of processing activities (Art. 4a(1)(a) and (f)) documentation
- Access control documentation for special-category data, listing the authorised persons and their confidentiality obligations (Art. 4a(1)(c)) documentation
- Retention period statement for special-category data processed for bias detection (Art. 4a(1)(e)) documentation
- Data governance and management practices operated across the lifecycle for the training, validation and testing data sets (Art. 10(2)) process
- Bias examination repeated when a data set changes and, where data outputs influence inputs for future operations, at each operation cycle (Art. 10(2)(f)) process
- Bias detection, prevention and mitigation measures applied and maintained as identified biases change (Art. 10(2)(g)) process
- Data gap and shortcoming identification repeated at each data set update (Art. 10(2)(h)) process
- Deletion of special-category data at the earlier of bias correction and the end of the retention period (Art. 4a(1)(e)) process
- Where a deployer or a provider of another AI system processes special-category data under Art. 4a(2), application of all the Art. 4a(1) conditions and safeguards to that processing (Art. 4a(2)) process
Art. 26Obligations of deployers of high-risk AI systems
- The instructions for use received from the provider, with the intended purpose and the conditions of use identified (Art. 26(1)) documentation
- The assignment record naming the natural persons who hold human oversight, with their competence, training and authority recorded (Art. 26(2)) documentation
- The log retention policy stating the retention period under Art. 26(6), the scope of logs under the deployer's control, and, for financial institutions, the place of the logs in the documentation kept under Union financial services law (Art. 26(6)) documentation
- The record that workers' representatives and the affected workers were informed before the system was put into service or used at the workplace (Art. 26(7)) documentation
- The EU database registration record under Art. 49 for a deployer that is a public authority or a Union institution, body, office or agency (Art. 26(8)) documentation
- The data protection impact assessment under Art. 35 of Regulation (EU) 2016/679 or Art. 27 of Directive (EU) 2016/680 that uses the Art. 13 information (Art. 26(9)) documentation
- For post-remote biometric identification: the authorisation requests, the police file record of each use, and the annual reports to the market surveillance and data protection authorities (Art. 26(10)) documentation
- The notice text that informs natural persons that they are subject to the use of the high-risk AI system (Art. 26(11)) documentation
- Technical and organisational measures for use in accordance with the instructions for use are maintained over the life of the deployment, including after a provider update to the instructions (Art. 26(1)) process
- Training and support for the persons assigned to human oversight are maintained and renewed when the system or the assigned persons change (Art. 26(2)) process
- Where the deployer controls the input data, the input data is checked against the intended purpose at each change of data source or population (Art. 26(4)) process
- The operation of the system is monitored on the basis of the instructions for use, and the provider is informed under Art. 72 where relevant (Art. 26(5)) process
- A risk within the meaning of Art. 79(1) triggers notification of the provider or distributor and the market surveillance authority without undue delay and suspension of use (Art. 26(5)) process
- A serious incident triggers immediate notification of first the provider, then the importer or distributor and the market surveillance authorities, with Art. 73 applied where the provider cannot be reached (Art. 26(5)) process
- For financial institutions, the monitoring duty is fulfilled through the internal governance arrangements, processes and mechanisms under Union financial services law (Art. 26(5)) process
- A deployer that is a public authority checks registration in the EU database before use, does not use an unregistered system, and informs the provider or the distributor (Art. 26(8)) process
- For post-remote biometric identification: authorisation is requested ex ante or no later than 48 hours, use stops and personal data is deleted on rejection, each use is limited to what is strictly necessary, and no adverse legal decision is based solely on the output (Art. 26(10)) process
- The notice to natural persons is delivered at the point where a decision is made or assisted (Art. 26(11)) process
- The deployer cooperates with competent authorities in any action they take in relation to the system (Art. 26(12)) process
A dataset card per data set carries the design choices, the collection and origin record, the preparation record and the assumptions statement; the Annex IV section 2(d) text serves the same purpose where it exists. The records of processing activities and the access control configuration for special-category data are supplied as an extract, together with a data flow diagram that shows every party with access.
Questions
What does ISO/IEC TR 24027:2021 provide for bias testing under the AI Act?
ISO/IEC TR 24027:2021 is a published technical report on bias in AI systems and AI-aided decision making. The report supplies the bias source categories (human cognitive, data and engineering bias), the lifecycle stages where each source enters, and candidate fairness metrics built on the confusion matrix. PR-10-BIAS uses it for the bias source register, the metric selection record, the disaggregated confusion matrices and the root-cause attribution of each failed finding. The report lists metrics; it states no acceptable disparity and no pass level.
What is the status of ISO/IEC 24027 under the AI Act?
ISO/IEC 24027 is a published technical report, ISO/IEC TR 24027:2021, from ISO/IEC JTC 1/SC 42. ISO/IEC TR 24027:2021 is not a harmonised standard, and no harmonised standard for Art. 10 is cited in the Official Journal, so no presumption of conformity attaches to a test built on it. The JTC 21 drafts prEN 18283 and prEN 18284 cover bias management and dataset quality, and both are at draft stage. A result against any of these documents is evidence for the technical documentation under Art. 11 and Annex IV.
How does ISO IEC TR 24027 set the fairness threshold for a test?
ISO/IEC TR 24027:2021 sets no fairness threshold. The report lists candidate fairness metrics and describes bias sources; it states no acceptable disparity. Art. 10 sets no numeric threshold. Under PR-10-BIAS the provider states a disparity target for each selected metric and justifies it against the context of use. Step 10 of the protocol records the target, the justification and its source level; step 33 compares each measured disparity with that target and marks the result pass or fail. A sector rule the provider adopts is recorded with its source.
Every inquiry receives a reply within 24 hours. Scope and availability are discussed per request.