Our model version changed: what re-testing does the AI Act require
A retrained or re-versioned model inside a high-risk AI system raises two questions under Regulation (EU) 2024/1689. Art. 15(1) requires consistent accuracy, robustness and cybersecurity throughout the lifecycle, so the affected tests are re-run on the new version. Art. 3(23) and Art. 43(4) decide whether the change is a substantial modification that starts a new conformity assessment. Art. 15 and Art. 43(4) apply to Annex III systems from the Annex III application date Deferred. This page gives the sequence, the evidence and the consequence of leaving the change untested.
Where this starts
The model has been retrained on new data, fine-tuned, swapped for a newer architecture, or re-deployed with a changed inference configuration, and the accuracy, robustness and adversarial reports in the technical file name the previous version hash. A provider of an Annex III system carries the Art. 15 requirement that the system performs consistently in those respects throughout its lifecycle, and the obligation record lists re-measurement after each model, data or operating-environment change as a process element of Art. 15(1). A test report holds for the version, the test sets and the conditions it was produced under; the lifecycle performance monitoring record written at the end of PR-15-ROB v1.0 names the changes that invalidate each report. A version change is the first trigger on that list.
The second question is whether the change is a substantial modification. Art. 3(23) defines one as a change after placing on the market or putting into service that was not foreseen in the initial conformity assessment and that either affects whether the system meets Chapter III, Section 2 or changes the intended purpose the system was assessed for. Art. 43(4) requires a new conformity assessment after a substantial modification, and it excludes one case: for a system that continues to learn after placing on the market, a change the provider pre-determined at the initial assessment and described in Annex IV section 2(f) is not substantial. The decision sits with the provider, its notified body where one is involved, and its counsel; a re-test supplies the measured evidence the decision rests on. Where a distributor, importer, deployer or other third party makes a substantial modification to a high-risk system already on the market, Art. 25(1)(b) makes that party the provider of the modified system.
The quality management system holds the procedure that runs between the two questions. Art. 17(1)(a) requires the strategy for regulatory compliance to include procedures for the management of modifications, and Art. 17(1)(d) requires examination, test and validation procedures with a stated frequency. The obligation record lists, as a common finding, modification management with no rule for deciding whether a change is substantial, so that the re-assessment step is never triggered. For a manufacturer of an Annex I product, the change enters the sectoral conformity route under Art. 43(3) from the Annex I application date Deferred: the Chapter III, Section 2 requirements are part of that assessment, and the change follows the change-notification rules of the sectoral law and the notified body under it. For a general-purpose AI model with systemic risk, the version change re-runs the Art. 55(1) evaluation and adversarial testing from the GPAI application date In force, and the mitigation effectiveness element under Art. 55(1)(b) is re-measured with the same evaluation and attack set.
No numeric threshold exists for any of this in the Act. The pass criterion for a re-test is the level the provider declared in the instructions for use under Art. 15(3) for the baseline version. Where the change moves that level, the target justification step of PR-15-ROB v1.0 records the new target and its source: a harmonised standard cited in the Official Journal, of which none is cited as of the anchor date, so no presumption of conformity exists; a JTC 21 draft such as prEN 18229-2Draft for the metric categories; an ISO/IEC document such as ISO/IEC TR 24029-1:2021Published for the method categories; a Commission guideline or code of practice; NIST AI RMF 1.0Published or a sector framework; or the state of the art the provider justifies. The re-test uses the same method category as the baseline for each robustness property, and a formal property shown under ISO/IEC 24029-2:2023Published for the previous weights does not carry over: it is re-established over the same bounded input region for the new version.
The sequence
The work runs in the order below, from the record of the change to the re-issued evidence pack, with the classification question answered before any test is re-run.
Record the new version hash and every change to weights, data, configuration or operating environment.
Record Environment manifest for the new version with the model version hash, inference configuration, seeds and tool versions
[PR-15-ROB step 1]
Classify each change against Art. 3(23), with the Annex IV section 2(f) pre-determined changes checked first.
Record Modification register entry with the classification as pre-determined, other or substantial, and the follow-on step
[Art. 3(23) and Art. 43(4); PR-43-MOCK steps 30 to 31]
Check that the modification record for the release follows the written modification management procedure.
Record Modification record sample with the substantial-modification decision and the conformity assessment step per version
[Art. 17(1)(a); PR-17-QMS step 10]
Read the re-test triggers in the lifecycle performance monitoring record and mark the invalidated reports.
Record List of the reports the change invalidates and the testable elements to re-run
[Art. 15(1); PR-15-ROB step 45]
Extract the declared levels from the instructions for use and record any target the change moves.
Record Declared-level extract and target justification record with the source level of any moved target
[Art. 15(3); PR-15-ROB steps 3 and 9]
Re-run the invalidated held-out, shifted-set, perturbation, formal, fault-injection and attack tests on the new version.
Record Run logs per test with the new version hash and the fixed test set hashes
[PR-15-ROB steps 14 to 30]
Run the new version and each released version under the same configuration and compare the metrics.
Record Consistency table across the released versions with the difference per metric
[Art. 15(1); PR-15-ROB steps 15 and 34]
Compare each result with the declared level and the baseline value and record the outcome.
Record Declared-level comparison table with meets or does not meet and the delta from the baseline value
[Art. 15(3); PR-15-ROB step 33]
Re-issue the reports and the evidence-pack index, and route a substantial modification to the Art. 43(4) procedure.
Record Updated traceability matrix, evidence-pack index with invalidated evidence marked, and re-test triggers for the new version
[Art. 43(4); PR-15-ROB step 45; PR-43-MOCK steps 31 and 38]
Where a declared level moved, the update to the instructions for use is a consistency check under PR-11-DOC v1.0, and a data change that touched the training set adds a bias re-measurement under PR-10-BIAS v1.0; neither is part of this sequence.
Evidence this situation ends with
| Artifact | Answers | Produced by |
|---|---|---|
| Modification register (Art. 43(4)) | Art. 3(23) and Art. 43(4): each change since the last assessment, classified as pre-determined, other or substantial, with the re-assessment trigger | PR-43-MOCK v1.0 |
| Regulatory compliance strategy and modification management procedure (Art. 17(1)(a)) | Art. 17(1)(a): the written procedure for the management of modifications and the conformity assessment step it names | the provider |
| Accuracy test report on held-out and distribution-shifted sets (PR-15-ROB output) | Art. 15(1) and 15(3): accuracy on the new version at or above the declared level, with the consistency table across released versions | PR-15-ROB v1.0 |
| Robustness test report with perturbation, fault-injection and fail-safe results (PR-15-ROB output) | Art. 15(4): outputs of the new version within the declared performance under perturbation, injected faults and fail-safe takeover | PR-15-ROB v1.0 |
| Adversarial and cybersecurity test report covering the Art. 15(5) attack list (PR-15-ROB output) | Art. 15(5): the outcome per attack class for the new version, with the threat model and the success criterion per attack | PR-15-ROB v1.0 |
| Accuracy declaration in instructions for use (Art. 15(3) and Art. 13(3)(b)(ii)) | Art. 15(3): the declared levels and metrics for the released version, updated where a level moved | the provider |
| Examination, test and validation procedure with stated frequency and the records of each run (Art. 17(1)(d)) | Art. 17(1)(d): the test record for the new version at the stated frequency, naming the version tested | the provider |
| Lifecycle performance monitoring record with re-test triggers | Art. 15(1): the model, data and operating-environment changes that invalidate each re-issued report | PR-15-ROB v1.0 |
Each report follows the test report specification, the modification register and the re-issued index follow the assessment memo and traceability matrix specifications, and a synthetic sample of each type is on the deliverables page. Seven of the artifacts are the ones the Art. 15 and Art. 17 obligation records list as what an assessor asks for; the modification register is the PR-43-MOCK output that records the classification.
If nothing is done
A high-risk AI system whose technical file rests on reports for a previous version has no evidence that the version on the market meets Art. 15(1), and a substantial modification without a new conformity assessment leaves the system on the market outside the procedure Art. 43(4) requires. Under Art. 79(2), a market surveillance authority that finds the system does not meet the Chapter III, Section 2 requirements requires the provider to take corrective action, to withdraw the system or to recall it within a period the authority sets, and Art. 99(4) makes failure to meet the provider obligations in Art. 16 subject to administrative fines. The action that follows from the provider's side is the sequence above: classify the change, re-run the invalidated tests on the released version, re-issue the evidence pack, and, where the change is substantial, start the Art. 43(4) procedure before the modified system is placed on the market or kept in service.
Services, guides and tools for this situation
The regression testing service runs the sequence above within the limits at what we cannot do: the substantial-modification assessment memo is a reasoned technical assessment for discussion with the provider's counsel and its notified body; the provider, its notified body and its counsel hold the Art. 43(4) decision, and the checklist tool lists the Art. 15 and Art. 17 test items for the Annex III provider profile, and the Annex III provider hub collects the pages for that role.
- Guides
Every inquiry receives a reply within 24 hours. Scope and availability are discussed per request.