Robustness and adversarial testing
We test high-risk AI systems against the accuracy, robustness and cybersecurity requirements of Art. 15 and general-purpose AI models with systemic risk against the adversarial testing duty of Art. 55(1), under PR-15-ROB v1.0. The attack set follows the five Art. 15(5) classes, tagged to the published OWASP Top 10 for LLM Applications and MITRE ATLAS. The provider receives four test reports for the technical file. Art. 15 applies to Annex III systems from the Annex III date Deferred; the limits are on what we cannot do.
Deliverables
The set is designed to satisfy the validation and testing evidence an assessor asks for under Annex IV sections 2(g) and 2(h) for Art. 15, and the adversarial testing documentation under Art. 55(1)(a) for a model with systemic risk. Every item is a test report; the specification and a synthetic sample of the type are on the deliverables page.
| Deliverable | Type | Format | Designed to satisfy |
|---|---|---|---|
| Accuracy report on held-out and distribution-shifted sets with declared metrics for instructions for use | Test report | PDF with Markdown source; CSV result tables; JSON run manifest | |
Robustness report: perturbation, noise, and edge-case results per ISO/IEC TR 24029-1 categoriesISO/IEC TR 24029-1:2021Published | Test report | PDF with Markdown source; CSV result tables; JSON run manifest | |
| Adversarial and red-team report: prompt injection, jailbreak, evasion, model extraction and poisoning attempts, mapped to OWASP LLM Top 10 and MITRE ATLAS | Test report | PDF with Markdown source; CSV result tables; JSON run manifest | |
| Feedback-loop test results for continuously learning systems | Test report | PDF with Markdown source; CSV result tables; JSON run manifest |
Who this is for
- Annex III providers
- Annex I manufacturers
- GPAI providers
A provider of an Annex III system carries the Art. 15 requirement for an appropriate level of accuracy, robustness and cybersecurity, and the Act sets no numeric threshold for any of the three. The provider sets and justifies the target, and an assessor checks the justification and the evidence behind it. A manufacturer embedding AI in an Annex I product already runs a technical file and a notified body relationship under the sectoral law, so the AI-specific test results are written to sit inside that file without repeating the sectoral tests. A provider of a general-purpose AI model above the Art. 51 systemic-risk threshold carries the Art. 55(1)(a) duty to evaluate the model and to conduct and document adversarial testing, and the Safety and Security chapter of the GPAI Code of PracticePublished describes what that evidence looks like.
The applicable dates are the regulation entries behind the urgency line, each with its verified source:
Scope
The engagement covers one AI system or model at a fixed version, recorded in an environment manifest with the version hash, the inference configuration and the random seeds. For a high-risk AI system the seven testable elements of Art. 15 are in scope: accuracy on a held-out set with the metrics declared under Art. 15(3), accuracy on a distribution-shifted set and across released versions, behaviour under perturbation, noise and malformed input, fail-safe takeover under an injected fault, feedback-loop drift for a system that continues to learn, resistance to each attack class in Art. 15(5), and the detection and response path for a simulated unauthorised attempt to alter outputs. The residual-risk and measure-activation tests that Art. 9 requires as results in the risk management file run under the same protocol, and the table below lists them. For a model with systemic risk, the engagement covers the evaluation per systemic risk, the adversarial campaign, the mitigation effectiveness delta and the model cybersecurity attempts under Art. 55; the Art. 51(2) training compute gate is a PR-CLS v1.0 record taken as input.
ISO/IEC TR 24029-1:2021Published supplies the method categories, statistical, formal and empirical, and ISO/IEC 24029-2:2023Published supplies the formal-property procedure over a bounded input region. The metric categories, the declaration structure and the perturbation classes follow prEN 18229-2Draft, and the AI-specific threat list behind Art. 15(5) follows prEN 18282Enquiry. Both JTC 21 documents are drafts: they are evidence of the state of the art, and no presumption of conformity attaches to them because no harmonised standard is cited in the Official Journal as of the anchor date shown on the standards page. Each attack in the adversarial set carries an entry from the published OWASP Top 10 for LLM Applications and a technique id from the published MITRE ATLAS knowledge base, so that the outcome matrix can be read against those references without translation.
Out of scope are the documentation and process elements of the same articles. The content of the accuracy declaration, the cybersecurity risk assessment and the redundancy description are inputs; PR-11-DOC v1.0 checks their content and consistency. Subgroup and bias measurement belong to PR-10-BIAS v1.0. Re-measurement after a model, data or environment change is the regression testing service. The protocol does not test the physical infrastructure of a model under Art. 55(1)(d), the serious incident path under Art. 55(1)(c), the hosting platform beyond the AI-specific attack surface, or testing in real-world conditions under Art. 60. The engagement is testing; a conformity assessment under Art. 43 is a separate procedure. Each result is a measured value next to a declared level, a prior defined threshold or a recorded acceptance criterion. The judgement whether that level is the appropriate one under Art. 15(1) belongs to the provider, its notified body and its counsel.
What we test
The table lists every testable element of the covered obligations with the protocol that runs it and the artifacts it produces for the record; documentation and process elements are listed in the inputs section below.
| Obligation | Testable element | Protocol | Artifacts |
|---|---|---|---|
| Art. 9 | The system performs consistently for its intended purpose across the operating conditions recorded in the risk management file, with a result per condition (Art. 9(6)). | PR-15-ROB v1.0 |
|
| Each test in the risk management file runs against a prior defined metric and probabilistic threshold, and the result either meets the threshold or does not (Art. 9(8)). | PR-15-ROB v1.0 | ||
| The residual risk of each hazard after the adopted measures is measured and compared with the acceptance criterion recorded for that hazard, and the overall residual risk is compared with the overall criterion (Art. 9(5)). | PR-15-ROB v1.0 | ||
| Each technical risk management measure adopted under Art. 9(2)(d), such as an input check, a filter, a fallback or a guardrail, activates on the hazard scenarios it addresses and produces the recorded effect (Art. 9(5)(a) and 9(5)(b)). | PR-15-ROB v1.0 | ||
| The system's behaviour under each reasonably foreseeable misuse condition recorded under Art. 9(2)(b) is observed and recorded against the expected behaviour. | PR-15-ROB v1.0 | ||
| Where the Art. 9(9) consideration identifies a likely adverse impact on persons under the age of 18 or other vulnerable groups, the test set contains cases that represent those groups and the results are reported for them. | PR-15-ROB v1.0 | ||
| Art. 15 | Accuracy on a held-out test set, measured with the metrics declared in the instructions for use, is at or above the declared level (Art. 15(1) and 15(3)) | PR-15-ROB v1.0 |
|
| Accuracy holds on a distribution-shifted set and across the released model versions, so that the system performs consistently throughout its lifecycle (Art. 15(1)) | PR-15-ROB v1.0 | ||
| Outputs stay within the declared performance under input perturbation, noise, malformed input, edge cases and inconsistent signals from natural persons or connected systems (Art. 15(4), first subparagraph) | PR-15-ROB v1.0 | ||
| The technical redundancy, backup or fail-safe solution takes over and holds the declared behaviour when a component fault is injected, where such a solution exists (Art. 15(4), second subparagraph) | PR-15-ROB v1.0 | ||
| For a system that continues to learn, feeding its own possibly biased outputs back as inputs does not move performance beyond the declared level, and the mitigation measure activates as documented (Art. 15(4), third subparagraph) | PR-15-ROB v1.0 | ||
| The system resists each attack class named in Art. 15(5), third subparagraph: data poisoning of the training set, model poisoning of pre-trained components, adversarial examples or model evasion, confidentiality attacks such as model extraction and membership inference, and exploitation of model flaws, with prompt injection included for systems built on a language model | PR-15-ROB v1.0 | ||
| The cybersecurity measures detect, respond to, resolve and control a simulated unauthorised attempt to alter the use, outputs or performance of the system (Art. 15(5), first and third subparagraphs) | PR-15-ROB v1.0 | ||
| Art. 55 | Model evaluation under Art. 55(1)(a): for each identified systemic risk, an evaluation is run with a named protocol and tool set that reflects the state of the art, and the result is recorded against a stated acceptance criterion | PR-15-ROB v1.0 |
|
| Adversarial testing under Art. 55(1)(a): a red-team campaign with defined attack classes, elicitation methods and a success criterion per attack is executed against the model, and each outcome is recorded | PR-15-ROB v1.0 | ||
| Mitigation effectiveness under Art. 55(1)(b): each safety mitigation is re-tested with the same evaluation and attack set, and the change from the pre-mitigation baseline is measured | PR-15-ROB v1.0 | ||
| Model cybersecurity under Art. 55(1)(d): unauthorised access to model weights, model extraction through the serving interface, training-data poisoning and prompt injection are attempted under a documented threat model, and the result of each attempt is recorded | PR-15-ROB v1.0 | ||
| Systemic-risk classification gate under Art. 51(2): cumulative training compute is calculated from the training records in floating point operations and compared with the presumption in Art. 51(2), with the calculation method recorded | PR-15-ROB v1.0 |
Regulation (EU) 2024/1689 sets no numeric threshold in Art. 15 or Art. 55, so every pass criterion comes from a six-level hierarchy. Level one is a harmonised standard cited in the Official Journal, and none is cited as of the anchor date, so no presumption of conformity exists for any criterion. Level two is a JTC 21 draft, stated with its stage: prEN 18229-2Draft for metric categories and prEN 18282Enquiry for the threat list, neither of which publishes a numeric pass level. Level three is an ISO/IEC document such as ISO/IEC 24029-2:2023Published, which supplies result semantics rather than a number. Level four is a Commission guideline or code of practice; for a model with systemic risk the provider sets its own acceptance criteria in the Safety and Security Framework under the GPAI Code of PracticePublished. Level five is NIST AI RMF 1.0Published or a sector framework, used for the metric selection rationale. Level six is the state of the art, justified by the provider. Where no published threshold exists, the target-justification step of PR-15-ROB v1.0 records the provider's target, its source level and the reason it fits the context of use, and the report shows the measured value next to that target.
Protocols used
One versioned protocol runs the engagement; its page states every step with the standard clause or Act provision it derives from.
- PR-15-ROB v1.0Robustness and adversarial testing protocolv1.0
PR-15-ROB v1.0 covers the testable elements of Art. 15, Art. 55 and the test-result elements of Art. 9 and Art. 13 in four phases: preparation, execution, analysis and reporting. It produces the accuracy, robustness, adversarial and feedback-loop reports listed above, each with the environment manifest, the test-set provenance and the declared-level comparison. It does not show that a declared level is the appropriate one. A statistical robustness estimate carries its confidence interval and sample size and is not a proof. A formal result holds only over the stated input region, and an undetermined result is recorded as undetermined and is not a pass. An adversarial result holds for the attack methods, budgets and access levels in the attack set, and the protocol cannot show the absence of attacks outside that set.
Engagement shape
The engagement is fixed-scope and scoped per request; the readiness assessment is the entry service and fixes which obligations and which system versions are in scope before this one starts. The provider supplies model access at a fixed version, the training, validation and test data with lineage, the instructions for use with the declared levels, Annex IV sections 2(g) and 2(h), the risk management file, the cybersecurity risk assessment with the Art. 15(5) applicability decisions, and, where they exist, the redundancy description, the continuous-learning design description and the version history. For a model with systemic risk the provider also supplies the Safety and Security Framework with its risk scenarios and, where the threat model puts them in scope, read access to the weight store and the training repository.
The work follows the order of the protocol. Preparation writes the environment manifest, records the classification gate, fixes the metrics and targets with their source level, assembles the held-out and distribution-shifted sets with hashes, and records the threat model and the attack set with a success criterion per attack. Execution runs the accuracy, perturbation, formal, fault-injection, feedback-loop and adversarial tests against the fixed version. Analysis compares each measured value with its declared level, threshold or acceptance criterion and records meets or does not meet. Reporting writes the four reports and the re-test triggers that invalidate them.
What we cannot do
- We do not perform conformity assessments where a notified body is required.
- We do not issue CE marking or the EU Declaration of Conformity.
- We are not a notified body.
- We do not give legal opinions on classification or liability.
- We test against requirements, prepare evidence, and structure documentation; compliance decisions remain with the provider or deployer, their notified body, and their counsel.
For this service the limit means that a test report states a measured value next to a declared level; it does not state that the level is appropriate, and it is not a conformity assessment under Art. 43. Where the provider records an Art. 15(5) attack class as not appropriate in its cybersecurity risk assessment, the protocol records that decision and does not test the class.
What the client must have ready
The documentation and process elements of the covered obligations are inputs to the test; the list below is generated from the obligation records.
Art. 9Risk management system
- The risk management system is documented in a risk management file that records the risks identified, the estimation and evaluation method, the measures adopted and the residual risk judgement (Art. 9(1) and 9(5)). documentation
- Each hazard is recorded with the intended purpose and the reasonably foreseeable misuse conditions under which it was identified and evaluated (Art. 9(2)(a) and 9(2)(b)). documentation
- For each risk, the file states whether design, development or adequate technical information can reasonably mitigate or eliminate it, which fixes the scope of Art. 9 for that risk (Art. 9(3)). documentation
- The interaction analysis records how the measures taken under Art. 10 to Art. 15 affect each other and the balance the provider chose (Art. 9(4)). documentation
- The acceptance criterion for the residual risk of each hazard and for the overall residual risk is stated before testing, with its rationale (Art. 9(5)). documentation
- The Art. 13 information and any deployer training are described together with the technical knowledge, experience, education and training expected of the deployer and the presumable context of use (Art. 9(5)(c)). documentation
- Where testing in real-world conditions takes place, the plan and the conditions under Art. 60 are documented (Art. 9(7)). documentation
- The prior defined metrics and probabilistic thresholds are recorded before the test runs, with the reason each is appropriate to the intended purpose (Art. 9(8)). documentation
- Test records identify the system version tested and show that a full test run took place before placing on the market or putting into service (Art. 9(8)). documentation
- The consideration of adverse impact on persons under the age of 18 and other vulnerable groups is recorded with its outcome (Art. 9(9)). documentation
- Where the provider combines Art. 9 with a risk management procedure under other Union law, a mapping shows where each aspect of Art. 9(1) to 9(9) is covered (Art. 9(10)). documentation
- The risk management system is established, implemented and maintained as a continuous iterative process that is planned and run through the entire lifecycle of the system (Art. 9(1) and 9(2)). process
- The risk management system is reviewed and updated regularly and systematically, with a defined trigger and a record of each review (Art. 9(2)). process
- Data from the post-market monitoring system under Art. 72 is analysed and other risks that arise are evaluated in the risk management cycle (Art. 9(2)(c)). process
- Risk management measures are adopted for the identified risks in the order of Art. 9(5): elimination or reduction by design and development, then mitigation and control measures, then information and training for deployers (Art. 9(2)(d) and 9(5)). process
- Testing takes place at any time during the development process as appropriate, and is repeated before placing on the market or putting into service (Art. 9(8)). process
- A provider subject to internal risk management requirements under other Union law may run the Art. 9 aspects inside that procedure (Art. 9(10)). process
Art. 15Accuracy, robustness and cybersecurity
- Declaration of the accuracy levels and the relevant accuracy metrics in the instructions for use, with the metric definitions, the test-set description and the measurement conditions (Art. 15(3)) documentation
- Rationale for the level of accuracy, robustness and cybersecurity chosen as appropriate for the intended purpose, with the source of each target (Art. 15(1)) documentation
- Description of the technical and organisational measures taken for resilience to errors, faults and inconsistencies (Art. 15(4), first subparagraph) documentation
- Description of the technical redundancy solution and the backup or fail-safe plan where one is used (Art. 15(4), second subparagraph) documentation
- Design description of how a continuously learning system limits feedback loops and the mitigation measures that address them (Art. 15(4), third subparagraph) documentation
- Cybersecurity risk assessment that shows the technical solutions are appropriate to the relevant circumstances and the risks, and that records, for each attack class in Art. 15(5), whether a measure applies and the reason where one is omitted as not appropriate (Art. 15(5), second and third subparagraphs) documentation
- Re-measurement of accuracy, robustness and cybersecurity after each model, data or operating-environment change, with a defined trigger, so that consistent performance is shown across the lifecycle (Art. 15(1)) process
- Operation of the organisational measures for resilience: fault handling, incident escalation and the operating procedures that respond to errors and inconsistencies in service (Art. 15(4), first subparagraph) process
- Operation of the feedback-loop mitigation measures while the system continues to learn, with periodic review of learned behaviour (Art. 15(4), third subparagraph) process
- Ongoing prevention, detection, response, resolution and control of AI-specific attacks: vulnerability management, threat monitoring and the response path for a detected attack (Art. 15(5)) process
Art. 55Obligations of providers of general-purpose AI models with systemic risk
- Adversarial testing documentation under Art. 55(1)(a): the campaign design, the attack set, the elicitation methods, the results and the model version tested documentation
- Systemic risk assessment record under Art. 55(1)(b): the identified systemic risks at Union level, their sources across development, placing on the market and use, and the mitigations chosen for each documentation
- Serious incident documentation under Art. 55(1)(c): the relevant information about each serious incident and the possible corrective measures, in the form reported to the AI Office documentation
- Cybersecurity measures description under Art. 55(1)(d): the threat model and the controls for the model and for its physical infrastructure, including weight storage, training infrastructure and the serving path documentation
- Means of demonstration under Art. 55(2): adherence to a code of practice under Art. 56, or the alternative adequate means submitted for Commission assessment; where the provider follows the GPAI Code of Practice, Safety and Security chapter, the Safety and Security Framework and the Safety and Security Model Report are the documents that show it documentation
- Systemic risk mitigation under Art. 55(1)(b): mitigations are applied and reviewed across the model lifecycle, from development through placing on the market and use process
- Serious incident tracking and reporting under Art. 55(1)(c): incidents are tracked continuously and reported without undue delay to the AI Office and, as appropriate, to national competent authorities process
- Cybersecurity maintenance under Art. 55(1)(d): the protection of the model and its physical infrastructure is kept at an adequate level as threats and the model change process
- Confidential handling under Art. 55(3): information and documentation submitted under this Article, including trade secrets, is identified so that it receives the Art. 78 treatment process
The inputs take the form the protocol page specifies: an inference endpoint or container image with a version hash, dataset files with a dataset card per set, and digital documents for the declaration, the risk management file and the cybersecurity risk assessment. A missing input is recorded as missing in the report; the protocol does not substitute a value for it.
Questions
How is jailbreak resistance testing done for an LLM?
Under PR-15-ROB, jailbreak resistance is one attack class in the adversarial set for a system built on a language model, alongside prompt injection and system-prompt leakage. Preparation records each attack with its elicitation method, budget, access level and a success criterion. Execution runs the attack set against the fixed model version. The adversarial and red-team report records the outcome per attack in a matrix tagged to the Art. 15(5) class, the OWASP entry and the ATLAS technique, and the result holds only for the attacks in the recorded attack set.
What does LLM security cover under Art. 15(5)?
Art. 15(5) requires a high-risk AI system to be resilient against unauthorised third parties who try to alter its use, outputs or performance, with technical solutions that address data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks and model flaws where appropriate. For a language-model system, PR-15-ROB adds prompt injection, jailbreak and system-prompt leakage to the Art. 15(5) attack set. The adversarial and red-team report records the outcome for each class, and the cybersecurity risk assessment records where the provider treats a class as not appropriate.
What is the OWASP Top 10 for LLM Applications?
The OWASP Top 10 for LLM Applications is a published list of vulnerability categories for applications built on large language models, maintained by the OWASP GenAI Security Project. PR-15-ROB uses the list as a tagging reference; it is not a standard and sets no pass criterion. Entries such as Prompt Injection, Sensitive Information Disclosure, Data and Model Poisoning and System Prompt Leakage label each attack in the outcome matrix of the adversarial and red-team report. The pass criterion for each attack is the success criterion recorded in preparation.
What is MITRE ATLAS?
MITRE ATLAS is a published knowledge base of adversary tactics and techniques against AI systems, structured as a matrix with technique ids. PR-15-ROB uses the knowledge base as a tagging reference; it is not a standard and sets no pass criterion. Each attack in the adversarial set carries an ATLAS technique id next to its Art. 15(5) class, so that the adversarial and red-team report reads against the knowledge base directly. The attack set, budgets and access levels are recorded in preparation; the report holds only for the recorded attacks.
What does ISO/IEC TR 24029-1:2021 provide for a robustness test?
ISO/IEC TR 24029-1:2021 is a published technical report that gives an overview of robustness assessment methods for neural networks and sorts them into statistical, formal and empirical categories, with the terminology for stating a robustness goal and its test. PR-15-ROB uses the three categories to select the method for each robustness property and to structure the robustness report. The technical report publishes no numeric pass level; the pass criterion comes from the level the provider declares in the instructions for use or the threshold recorded in the risk management file.
Every inquiry receives a reply within 24 hours. Scope and availability are discussed per request.