+Artikel 60 Testen van AI-systemen met een hoog risico onder reële omstandigheden buiten AI-testomgevingen voor regelgeving
---+Artikel 60 1.
---+Artikel 60 2.
---+Artikel 60 3.
---+Artikel 60 4.
---+Artikel 60 5.
---+Artikel 60 6.
---+Artikel 60 7.
---+Artikel 60 8.
---+Artikel 60 9.
|
Artikel 60 Testen van AI-systemen met een hoog risico onder reële omstandigheden buiten AI-testomgevingen voor regelgeving
-
1. Overzicht
| Titel |
Regelgeving |
|
Artikel 60 1.
|
AI-systemen met een hoog risico kunnen onder reële omstandigheden buiten AI-testomgevingen voor regelgeving worden getest door aanbieders of potentiële aanbieders van in bijlage III vermelde AI-systemen met een hoog risico of van AI-systemen met een hoog risico die onder de in bijlage I, afdeling A, vermelde harmonisatiewetgeving van de Unie vallen, in overeenstemming met dit artikel en het in dit artikel bedoelde plan voor testen onder reële omstandigheden, onverminderd de verbodsbepalingen krachtens artikel 5. De Commissie specificeert, door middel van uitvoeringshandelingen, de precieze onderdelen van het plan voor tests onder reële omstandigheden. Die uitvoeringshandelingen worden vastgesteld volgens de in artikel 98, lid 2, bedoelde onderzoeksprocedure. Dit lid doet geen afbreuk aan Unie- of nationale wetgeving voor het testen onder reële omstandigheden van AI-systemen met een hoog risico die verband houden met producten die onder de in bijlage I vermelde harmonisatiewetgeving van de Unie vallen.
|
|
Artikel 60 2.
|
Aanbieders of potentiële aanbieders kunnen zelf of in samenwerking met een of meer gebruiksverantwoordelijken of potentiële gebruiksverantwoordelijken onder reële omstandigheden tests uitvoeren op in bijlage III bedoelde AI-systemen met een hoog risico of AI-systemen met een hoog risico die onder de in bijlage I, afdeling A, vermelde harmonisatiewetgeving van de Unie vallen, op elk moment vóór het in de handel brengen of in gebruik stellen van het AI-systeem met een hoog risico.
|
|
Artikel 60 3.
|
Het testen van AI-systemen met een hoog risico onder reële omstandigheden uit hoofde van dit artikel doet geen afbreuk aan de ethische toetsing die op grond van het Unie- of nationale recht is vereist.
|
|
Artikel 60 4.
|
Aanbieders of potentiële aanbieders mogen alleen testen onder reële omstandigheden als is voldaan aan alle volgende voorwaarden:
a)
de aanbieder of potentiële aanbieder heeft een plan voor tests onder reële omstandigheden opgesteld en ingediend bij de markttoezichtautoriteit in de lidstaat waar onder reële omstandigheden moet worden getest;
b)
de markttoezichtautoriteit in de lidstaat waar onder reële omstandigheden moet worden getest, heeft het testen onder reële omstandigheden en het plan voor tests onder reële omstandigheden goedgekeurd; indien de markttoezichtautoriteit binnen dertig dagen geen antwoord heeft gegeven, worden het testen onder reële omstandigheden en het plan voor tests onder reële omstandigheden geacht te zijn goedgekeurd; indien het nationale recht niet voorziet in een stilzwijgende goedkeuring, blijft het testen onder reële omstandigheden onderworpen aan een toestemming;
c)
de aanbieder of potentiële aanbieder, met uitzondering van aanbieders of potentiële aanbieders van in de punten 1, 6 en 7 van bijlage III bedoelde AI-systemen met een hoog risico op de gebieden rechtshandhaving, migratie, asiel en grenstoezichtsbeheer en AI-systemen met een hoog risico als bedoeld in punt 2 van bijlage III, heeft het testen onder reële omstandigheden geregistreerd overeenkomstig artikel 71, lid 4, met een Uniebreed uniek identificatienummer en de in bijlage IX gespecificeerde informatie; de aanbieder of potentiële aanbieder van in de punten 1, 6 en 7 van bijlage III bedoelde AI-systemen met een hoog risico op de gebieden van rechtshandhaving, migratie, asiel en grenstoezichtsbeheer, heeft het testen onder reële omstandigheden geregistreerd in het beveiligde niet-openbare gedeelte van de EU-databank overeenkomstig artikel 49, lid 4, punt d), met een Uniebreed uniek identificatienummer en de daarin gespecificeerde informatie; de aanbieder of potentiële aanbieder van in punt 2 van bijlage III bedoelde AI-systemen met een hoog risico heeft het testen onder reële omstandigheden geregistreerd overeenkomstig artikel 49, lid 5;
d)
de aanbieder of potentiële aanbieder die onder reële omstandigheden test, is in de Unie gevestigd of heeft een in de Unie gevestigde wettelijke vertegenwoordiger aangewezen;
e)
gegevens die zijn verzameld en verwerkt met het oog op het testen onder reële omstandigheden mogen alleen aan derde landen worden doorgegeven mits er passende en toepasselijke waarborgen uit hoofde van het Unierecht worden toegepast;
f)
het testen onder reële omstandigheden duurt niet langer dan nodig is om de doelstellingen ervan te verwezenlijken en in geen geval langer dan zes maanden, met een mogelijke verlenging van nog eens zes maanden indien de aanbieder of potentiële aanbieder de markttoezichtautoriteit daar vooraf van in kennis stelt, met een uitleg waarom een dergelijke verlenging noodzakelijk is;
g)
proefpersonen die onder reële omstandigheden worden getest en die tot kwetsbare groepen behoren vanwege hun leeftijd of handicap, worden naar behoren beschermd;
h)
indien een aanbieder of potentiële aanbieder het testen onder reële omstandigheden organiseert in samenwerking met een of meer gebruiksverantwoordelijken of potentiële gebruiksverantwoordelijken, worden zij geïnformeerd over alle aspecten van het testen die relevant zijn voor hun beslissing om deel te nemen, en krijgen zij de relevante gebruiksinstructies van het AI-systeem als bedoeld in artikel 13; de aanbieder of potentiële aanbieder en de gebruiksverantwoordelijke of potentiële gebruiksverantwoordelijke sluiten een overeenkomst waarin hun taken en verantwoordelijkheden worden gespecificeerd teneinde te waarborgen dat de bepalingen voor het testen onder reële omstandigheden uit hoofde van deze verordening en ander toepasselijk Unie- en nationaal recht worden nageleefd;
i)
de proefpersonen die onder reële omstandigheden worden getest, hebben geïnformeerde toestemming gegeven overeenkomstig artikel 61, of, in het geval van rechtshandhaving, indien het vragen om geïnformeerde toestemming het testen van het AI-systeem onder reële omstandigheden onmogelijk zou maken, de test zelf en het resultaat van de test onder reële omstandigheden hebben geen negatieve gevolgen voor de proefpersonen en hun persoonsgegevens worden na de uitvoering van test gewist;
j)
op het testen onder reële omstandigheden wordt daadwerkelijk toezicht gehouden door de aanbieder of potentiële aanbieder, alsook door gebruiksverantwoordelijken of potentiële gebruiksverantwoordelijken via personen die voldoende zijn gekwalificeerd op het relevante gebied en beschikken over de nodige capaciteiten, opleiding en bevoegdheden om hun taken uit te voeren;
k)
de voorspellingen, aanbevelingen of beslissingen van het AI-systeem kunnen daadwerkelijk worden teruggedraaid en genegeerd.
|
|
Artikel 60 5.
|
Proefpersonen van het testen onder reële omstandigheden of hun wettelijke vertegenwoordiger, al naargelang het geval, kunnen zich, zonder nadelige gevolgen en zonder enige rechtvaardiging, te allen tijde uit het testen terugtrekken door hun geïnformeerde toestemming in te trekken, waarna zij om de onmiddellijke en permanente verwijdering van hun persoonsgegevens kunnen verzoeken. De intrekking van de geïnformeerde toestemming heeft geen gevolgen voor de reeds uitgevoerde activiteiten.
|
|
Artikel 60 6.
|
Overeenkomstig artikel 75 verlenen de lidstaten hun markttoezichtautoriteiten de bevoegdheid om aanbieders en potentiële aanbieders te verplichten informatie te verstrekken, om onaangekondigde inspecties op afstand of ter plaatse uit te voeren en om toezicht te houden op de uitvoering van het testen onder reële omstandigheden en de aanverwante AI-systemen met een hoog risico. Markttoezichtautoriteiten gebruiken deze bevoegdheden om ervoor te zorgen dat het testen onder reële omstandigheden zich veilig kan ontwikkelen.
|
|
Artikel 60 7.
|
Elk ernstig incident dat tijdens het testen onder reële omstandigheden wordt vastgesteld, wordt gemeld bij de nationale markttoezichtautoriteit overeenkomstig artikel 73. De aanbieder of potentiële aanbieder neemt onmiddellijke risicobeperkende maatregelen of, bij gebreke daarvan, schorst het testen onder reële omstandigheden totdat dergelijke risicobeperkende maatregelen zijn getroffen, of beëindigt anders het testen. De aanbieder of potentiële aanbieder stelt een procedure vast voor het onmiddellijk terugroepen van het AI-systeem bij een dergelijke beëindiging van het testen onder reële omstandigheden.
|
|
Artikel 60 8.
|
Aanbieders of potentiële aanbieders stellen de nationale markttoezichtautoriteit in de lidstaat waar er onder reële omstandigheden moet worden getest, in kennis van de opschorting of beëindiging van het testen onder reële omstandigheden en van de eindresultaten.
|
|
Artikel 60 9.
|
De aanbieder of potentiële aanbieder zijn aansprakelijk op grond van het toepasselijke Unie- en nationale aansprakelijkheidsrecht voor schade die tijdens het testen onder reële omstandigheden wordt veroorzaakt.
|
,
1.1 Referenties
1.2 Geïdentificeerde eisen
1.3 Gerelateerde standaarden
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- AI TEVV Reporting (SCF)
- AI TEVV Reporting (SCF)
- Serious Incident Reporting For AI & Autonomous Technologies (SCF)
2. Geïdentificeerde eisen
3. Gerelateerde standaarden
Standaarden
| Bron |
Eis |
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
AI TEVV Reporting
Description
Mechanisms exist to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report AI testing results to management
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Document and communicate AI test results to stakeholders
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal AI TEVV reporting process ∙ Standardized reporting templates
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI TEVV reporting program ∙ Regular stakeholder briefings ∙ Findings tracking
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI TEVV reporting framework ∙ Automated test result dashboards ∙ Board-level AI risk reporting
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
AI TEVV Reporting
Description
Mechanisms exist to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report AI testing results to management
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Document and communicate AI test results to stakeholders
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal AI TEVV reporting process ∙ Standardized reporting templates
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI TEVV reporting program ∙ Regular stakeholder briefings ∙ Findings tracking
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI TEVV reporting framework ∙ Automated test result dashboards ∙ Board-level AI risk reporting
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Serious Incident Reporting For AI & Autonomous Technologies
Description
Mechanisms exist to report any serious incident involving operational Artificial Intelligence (AI) and Autonomous Technologies (AAT) to relevant authorities as to when and where the serious incident occurred, in accordance with mandated reporting timelines.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report serious AI incidents to relevant authorities as required
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Incident reporting procedure for serious AI incidents
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal serious AI incident reporting process ∙ Regulatory reporting contacts
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI incident response plan with regulatory reporting requirements ∙ Legal team involvement
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI incident response program ∙ Automated regulatory notification workflows ∙ Legal and compliance integration
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
Artificial Intelligence and Autonomous Technology (AAT) domain capabilities are ad hoc and inconsistent. Capability criteria associated with this control may include: ▪ Policies, standards & procedures associated with AAT domain capabilities provide limited coverage due to the depth and breadth of the existing documentation. ▪ AAT-related processes are expected to follow the organization's existing processes (e.g., incident response, asset management, change control and risk assessments). ▪ No formal Governance, Risk & Compliance (GRC) team exists to provide AAT oversight, where the Chief Information Officer (CIO), or similar function, governs technology decisions what is acceptable for AAT within the organization.
Level 2 Planned Tracked
Artificial Intelligence and Autonomous Technology (AAT) capabilities are requirements-driven, but are not standardized across the entity (e.g., local/regional level consistency). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are documented and maintained by process owners. ▪ IT and/or cybersecurity personnel work with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address applicable statutory, regulatory and/or contractual requirements for Technology Assets, Applications, Services and/or Data (TAASD). ▪ Artificial Intelligence (AI)-related controls are primarily administrative and preventative in nature (e.g., policies, standards and procedures & guidelines). ▪ Asset management may be a defined function (e.g., team or department) or assigned as an additional duty to existing IT and/or cybersecurity personnel.
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to report any serious incident involving operational AAT to relevant authorities as to when and where the serious incident occurred, in accordance with mandated reporting timelines.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|