QMS gap analysis protocol (PR-17-QMS)

PR-17-QMS v1.0 is built from prEN 18286Published, from ISO/IEC 42001:2023Published including its Annex A controls, from ISO/IEC 23894:2023Published for the risk management process, from prEN 18228Enquiry at enquiry stage for the Art. 9 record set, and from Art. 17 and Art. 9 of Regulation (EU) 2024/1689.

Scope

The protocol covers a high-risk AI system under Annex I or Annex III of Regulation (EU) 2024/1689 whose provider holds, or is building, the quality management system that Art. 17(1) names. The Art. 17 and Art. 9 obligations apply to Annex III systems from the Annex III application date 2027-12-02Deferred; Regulation (EU) 2026/1744 moved that date and left the content of both articles unchanged.

The protocol covers three groups of obligation elements:

• The written policies, procedures and instructions for each aspect in Art. 17(1) points (a) to (m), the proportionality rationale under Art. 17(2), and the sectoral or financial-services mapping under Art. 17(3) and 17(4). Each is checked as a document with a stated content.
• The eight testable elements of the Art. 17 obligation record: modification records, design verification records, quality control records, test and validation records, dataset records, the serious incident walk-through, the record retrieval test and the accountability walk-through.
• The risk management system under Art. 17(1)(g): the risk management file, the hazard register, the acceptance criteria, the prior defined metrics and thresholds under Art. 9(8), the residual risk comparison under Art. 9(5), and the post-market monitoring and serious incident procedures under Art. 17(1)(h) and (i). The six testable elements of the Art. 9 obligation record are checked as records in the risk management file.

The protocol does not execute the tests that the risk management file names; PR-15-ROB v1.0 executes them and this protocol records the cross-reference. The protocol does not perform a conformity assessment, does not decide whether a change is a substantial modification, and does not give an opinion on SME, start-up or SMC status. The QMS gap analysis service uses this protocol; the limits of that service are stated on what we cannot do.

Articles covered

The Art. 9 elements are checked here as records in the risk management file. Execution of each test sits in PR-15-ROB v1.0, and step 36 records the cross-reference.

Inputs required

Steps

Preparation

1. Record the system name, the version under review, the provider role and the Annex I or Annex III route.

   Record Review scope record

   [AIAssuranceLabs procedure]

2. Collect the written policies, procedures and instructions of the quality management system into a document inventory.

   Record Document inventory with document id, title, version and owner role

   [Regulation (EU) 2024/1689, Art. 17(1)]

3. Map each Art. 17(1) aspect, (a) to (m), to the clause of each published management-system standard.

   Record Clause map: aspect, clause of each published standard, Annex A control

   [prEN 18286, published, Annex ZA mapping to the Act]

4. Record the proportionality rationale and any SME, start-up or SMC status the provider claims.

   Record Proportionality record with the size status claimed and the elements simplified

   [Regulation (EU) 2024/1689, Art. 17(2) and Art. 63(1)]

5. Record each Art. 17(1) aspect the provider holds in a sectoral quality system or under financial services governance rules.

   Record Sectoral mapping with points (g), (h) and (i) marked as kept in scope

   [Regulation (EU) 2024/1689, Art. 17(3) and 17(4)]

6. Define the record sample for each aspect: the current version and one earlier version within the retention period.

   Record Sampling plan with the version ids and the retention period the procedure states

   [AIAssuranceLabs procedure]

7. Collect the internal audit programme, the audit reports and the management review minutes the provider holds.

   Record Audit evidence inventory

   [ISO/IEC 42001:2023 §9.2 and §9.3]

8. Open the gap register with one row per Art. 17(1) aspect and one row per Art. 9 paragraph.

   Record Gap register, empty, in the layout below

   [AIAssuranceLabs procedure]

row_id         : GR-17-01a
aspect         : Art. 17(1)(a) strategy for regulatory compliance, management of modifications
clause_map     : prEN 18286 clause on change control; ISO/IEC 42001:2023 A.2.2, A.6.1.2
element        : modification record per released version (testable)
evidence_id    : DOC-041 v3; CHG-2201; CHG-2207
method         : record sampling, current version and one earlier version
finding        : CHG-2207 carries no substantial-modification decision field
classification : partial
owner_role     : head of quality
action_ref     : CAP-03

Execution

Steps 9 to 23 walk the Art. 17(1) aspects in order. Steps 24 to 36 review the risk management system under Art. 17(1)(g).

9. Check that the compliance strategy names the conformity assessment procedure and the modification management procedure.

   Record Gap register row (a), documentation column

   [Regulation (EU) 2024/1689, Art. 17(1)(a)]

10. Sample the modification record of each released version and check the substantial-modification decision and the follow-on step.

    Record Modification record sample with the decision and the conformity assessment step per version

    [prEN 18286, published, clause on change control; Regulation (EU) 2024/1689, Art. 17(1)(a) and Art. 43(4)]

11. Sample the design verification records and trace each verified item to a design input.

    Record Design verification sample with the trace result per verified item

    [ISO/IEC 42001:2023 Annex A control A.6.2.4; Regulation (EU) 2024/1689, Art. 17(1)(b)]

12. Sample the quality control records of the current release and compare each check with the development procedure.

    Record Quality control sample: checks named by the procedure, checks found, outcomes

    [ISO/IEC 42001:2023 Annex A control A.6.1.3; Regulation (EU) 2024/1689, Art. 17(1)(c)]

13. Check that the test and validation procedure states each activity before, during and after development, with its frequency.

    Record Gap register row (d), documentation column

    [Regulation (EU) 2024/1689, Art. 17(1)(d)]

14. Sample the test and validation records against the stated frequency and check that each names the version tested.

    Record Test record sample: runs expected at the stated frequency, runs found, version id per record

    [Regulation (EU) 2024/1689, Art. 17(1)(d); ISO/IEC 42001:2023 Annex A control A.6.2.4]

15. Check the list of applied specifications and standards and the statement of means for each Section 2 requirement not covered.

    Record Applied standards list check with the coverage statement per Chapter III Section 2 requirement

    [Regulation (EU) 2024/1689, Art. 17(1)(e)]

16. Sample the dataset records of the training, validation and testing sets against the steps the data management procedure names.

    Record Dataset record sample: acquisition, labelling, storage and retention step per set

    [ISO/IEC 42001:2023 Annex A controls A.7.3 to A.7.6; Regulation (EU) 2024/1689, Art. 17(1)(f)]

17. Check that the post-market monitoring plan is in the system documentation and names the data collected and the analysis trigger.

    Record Post-market monitoring plan check: presence, data collected, analysis trigger

    [prEN 18286, published, clause on post-market monitoring; Regulation (EU) 2024/1689, Art. 17(1)(h) and Art. 72(3)]

18. Check that the serious incident procedure names the reporting path, the authority and the Art. 73 timeframes.

    Record Gap register row (i), documentation column

    [Regulation (EU) 2024/1689, Art. 17(1)(i) and Art. 73]

19. Walk one simulated serious incident through the procedure until the reporting step names the authority and the timeframe.

    Record Incident walk-through record: scenario, steps taken, authority identified, Art. 73 timeframe identified, stop point

    [prEN 18286, published, clause on serious incident handling; ISO/IEC 42001:2023 Annex A control A.8.4]

20. Check that the communication procedure covers authorities, notified bodies, other operators, customers and other interested parties.

    Record Gap register row (j), documentation column

    [Regulation (EU) 2024/1689, Art. 17(1)(j); ISO/IEC 42001:2023 Annex A control A.8.5]

21. Retrieve each sampled record for the current version and for one earlier version within the stated retention period.

    Record Retrieval test record: record id, version, found or not found, location

    [Regulation (EU) 2024/1689, Art. 17(1)(k) and Art. 18; ISO/IEC 42001:2023 §7.5.3]

22. Check that the resource management description names the resources and the security-of-supply measures.

    Record Gap register row (l), documentation column

    [ISO/IEC 42001:2023 Annex A control A.4.2; Regulation (EU) 2024/1689, Art. 17(1)(l)]

23. Walk through each responsibility in the accountability framework with the named role holder and record the confirmation.

    Record Role walk-through record: responsibility, role, holder confirmation, gap

    [ISO/IEC 42001:2023 §5.3 and Annex A control A.3.2; Regulation (EU) 2024/1689, Art. 17(1)(m)]

24. Check that the risk management file records the lifecycle plan, the review trigger and each review carried out.

    Record Gap register rows Art. 9(1) and 9(2), with the Art. 9(7) real-world testing plan and the Art. 9(10) mapping where used

    [ISO/IEC 23894:2023 §6.6 monitoring and review; Regulation (EU) 2024/1689, Art. 9(1) and 9(2)]

25. Check that the hazard register records each risk with its intended purpose and its reasonably foreseeable misuse condition.

    Record Hazard register check with the Art. 9(3) scope statement per risk

    [prEN 18228, enquiry stage, clause on risk identification; ISO/IEC 23894:2023 §6.4.2]

26. Check that the acceptance criterion for each hazard and for the overall residual risk is recorded and dated before testing.

    Record Acceptance-criteria record: hazard id, criterion, rationale, date recorded, first test date

    [ISO/IEC 23894:2023 §6.3.4 risk criteria; Regulation (EU) 2024/1689, Art. 9(5)]

27. Check that the estimation and evaluation method is stated and applied to each hazard in the register.

    Record Estimation and evaluation method check per hazard

    [ISO/IEC 23894:2023 §6.4.3 and §6.4.4; Regulation (EU) 2024/1689, Art. 9(2)(b)]

28. Check that a test record exists for each operating condition in the file with a result per condition.

    Record Operating condition coverage table: condition, test record id, result

    [Regulation (EU) 2024/1689, Art. 9(6)]

29. Check that each test record names a metric and threshold dated before the run, with the result marked.

    Record Prior-definition check: metric, threshold, date recorded, run date, met or not met

    [prEN 18228, enquiry stage, clause on risk evaluation records; Regulation (EU) 2024/1689, Art. 9(8)]

30. Check that each misuse condition has a test record with the observed behaviour against the expected behaviour.

    Record Misuse condition coverage table: condition, expected behaviour, observed behaviour, record id

    [ISO/IEC 23894:2023 Annex B risk sources; Regulation (EU) 2024/1689, Art. 9(2)(b)]

31. Check that each technical risk management measure has a verification record on its hazard scenarios with the recorded effect.

    Record Measure verification table: measure, hazard scenarios, activation record, effect recorded

    [ISO/IEC 23894:2023 §6.5.3 risk treatment plans; Regulation (EU) 2024/1689, Art. 9(5)(a) and 9(5)(b)]

32. Compare the recorded residual risk of each hazard, and the overall residual risk, with the recorded acceptance criterion.

    Record Residual risk comparison: hazard id, residual risk value, criterion, within or outside criterion; overall row

    [ISO/IEC 23894:2023 §6.4.4 risk evaluation; Regulation (EU) 2024/1689, Art. 9(5)]

33. Check the vulnerable-groups consideration record and, where impact is likely, the test cases and results for those groups.

    Record Vulnerable groups check: consideration outcome, groups named, test cases present, results reported

    [Regulation (EU) 2024/1689, Art. 9(9)]

34. Check that the post-market monitoring data analysis record feeds the risk evaluation of the current cycle.

    Record Post-market data feedback check: analysis record id, risk evaluation entry it changed

    [ISO/IEC 23894:2023 §6.6 monitoring and review; Regulation (EU) 2024/1689, Art. 9(2)(c)]

35. Check that the interaction analysis records how the measures under Art. 10 to Art. 15 affect each other.

    Record Interaction analysis check

    [Regulation (EU) 2024/1689, Art. 9(4)]

36. Record the PR-15-ROB report id that holds the execution result for each Art. 9 test element.

    Record Cross-reference table from each Art. 9 testable element to the risk-driven test report

    [AIAssuranceLabs procedure]

Analysis

37. Classify each gap register row as conforms, partial, gap or not applicable, with the evidence id.

    Record Gap register, classified, with an evidence id per row

    [AIAssuranceLabs procedure]

38. Record, for each aspect with no published frequency, period or depth, the provider's justification against the context of use.

    Record Element-evidence record: aspect, value set by the provider, justification, source consulted

    [AIAssuranceLabs procedure]

39. Map each open gap to the clause and Annex A control that would hold the corrective action.

    Record Gap-to-control map

    [ISO/IEC 42001:2023 §10.2 and Annex A]

40. Assign each open gap a corrective action, an owner role and the evidence that will show the action is closed.

    Record Corrective-action plan rows: gap row id, action, owner role, closing evidence

    [prEN 18286, published, clause on nonconformity and corrective action; ISO/IEC 42001:2023 §10.2]

41. Build the internal audit checklist from the clause map, with one question per clause and the evidence expected.

    Record Internal audit checklist, in the layout below

    [ISO/IEC 42001:2023 §9.2 internal audit]

check_id | clause                                   | question                                                             | evidence expected                          | result
IA-01    | prEN 18286, documentation requirements   | Are the policies, procedures and instructions written and versioned? | document inventory with version ids        | open
IA-07    | ISO/IEC 42001:2023 §9.2 internal audit   | Did the last internal audit cover every Art. 17(1) aspect?           | audit report with a clause coverage table  | open
IA-12    | Regulation (EU) 2024/1689, Art. 9(8)     | Is each metric and threshold dated before its first test run?        | acceptance-criteria record; test records   | open
IA-15    | ISO/IEC 42001:2023 A.8.4 incidents       | Does the incident procedure name the authority and the timeframe?    | incident procedure; walk-through record    | open

Reporting

42. Write the gap analysis report with the clause map, the classified register, the sample records and the walk-through records.

    Record QMS gap analysis report (PR-17-QMS output)

    [AIAssuranceLabs procedure]

43. Write the risk management system review record with the coverage tables and the residual risk comparison.

    Record Risk management system review record (PR-17-QMS output)

    [AIAssuranceLabs procedure]

44. Write the corrective-action plan and the internal audit checklist as separate documents with the register row ids.

    Record Corrective-action plan (PR-17-QMS output); Internal audit preparation checklist (PR-17-QMS output)

    [AIAssuranceLabs procedure]

45. State in each output the documents reviewed, the records sampled and the limits of the review.

    Record Limits statement in each output

    [AIAssuranceLabs procedure]

Thresholds and criteria

Regulation (EU) 2024/1689 sets no numeric threshold in Art. 17 or Art. 9. Art. 17(1) names the aspects a quality management system contains and names no frequency, sampling depth or retention period for any of them. Art. 9(8) requires testing against prior defined metrics and probabilistic thresholds and leaves the values to the provider, with the single condition that they are appropriate to the intended purpose. Art. 9(5) requires that residual risk is judged to be acceptable and gives no acceptance level. The pass criteria in this protocol come from the following levels of the source hierarchy; the standards page shows the stage of each document.

None of the documents at levels 2 to 5 publishes a numeric acceptance level for residual risk, a test frequency or a sampling depth. Step 26 (the acceptance-criteria step) records each acceptance criterion with its rationale and its date. Step 38 (the element-evidence step) records the provider's justification for each value the provider set.

Each step type has one pass rule:

• A documentation check passes when the document exists and states each content item the step names.
• A record sample passes when every sampled record is found and shows the content the written procedure names for it.
• A walk-through passes when the procedure reaches the end point the step names with each named item identified.
• A comparison passes when the recorded value sits within the provider's recorded criterion.

A pass is a statement about the documents and the sampled records, and no conformity opinion follows from it. The Art. 17 obligation page names the assessor and the notified body where one is required.

Outputs

The specification and a synthetic sample of each deliverable are on the deliverables page.

Limitations

• The protocol reviews documents and sampled records at one point in the lifecycle. It cannot show that the quality management system operated between the sampled versions or that it will operate after the review.
• The protocol checks that test records exist, that metrics and thresholds were dated before each run, and that results sit within the recorded criteria. It cannot show that a test result is correct; PR-15-ROB v1.0 executes the tests.
• A walk-through follows one simulated incident and one set of role holders. It cannot show that every incident type reaches the reporting step or that every role holder acts as the framework states.
• The classification of a change as a substantial modification, and the SME, start-up or SMC status under Art. 17(2) and Art. 63(1), are the provider's decisions. The protocol records them and does not decide them.
• The Art. 17(1)(e) check records the statement of means the provider gives for each Chapter III Section 2 requirement. It does not test whether the means are adequate.
• No harmonised standard for Art. 17 or Art. 9 is cited in the Official Journal as of the verified-on date, so no presumption of conformity attaches to any output of this protocol. The stage of prEN 18228Enquiry is enquiry; its clause content can change, and the record set in steps 25 and 29 follows the text at the verified-on date.
• The protocol applies to a system the provider has classified as high-risk. Where the provider relies on Art. 6(3) to conclude that an Annex III system is not high-risk, the registration duty that follows is an unsettled point and sits outside this protocol.

Sources

• prEN 18286Published, Quality management system for EU AI Act regulatory purposes, CEN-CENELEC JTC 21; approved as a European standard, published, not cited in the Official Journal.
• ISO/IEC 42001:2023Published, Artificial intelligence management system, ISO/IEC JTC 1/SC 42; published, with Annex A controls.
• ISO/IEC 23894:2023Published, Artificial intelligence risk management guidance, ISO/IEC JTC 1/SC 42; published.
• prEN 18228Enquiry, AI risk management, requirements and guidance, CEN-CENELEC JTC 21; at enquiry stage.
• NIST AI RMF 1.0Published, AI Risk Management Framework, NIST; published.
• Regulation (EU) 2024/1689, consolidated text on EUR-Lex, Art. 9, Art. 17, Art. 18, Art. 43, Art. 63, Art. 72 and Art. 73.

Revision history

Version 1.0 is the initial version; the anchor column shows the verified-on stamp of the regulation entry the revision names.