+Artikolu 60 Ittestjar ta' sistemi tal-IA b'riskju għoli f'kundizzjonijiet tad-dinja reali barra mill-ambjenti ta' esperimentazzjoni regolatorja għall-IA
---+Artikolu 60 1.
---+Artikolu 60 2.
---+Artikolu 60 3.
---+Artikolu 60 4.
---+Artikolu 60 5.
---+Artikolu 60 6.
---+Artikolu 60 7.
---+Artikolu 60 8.
---+Artikolu 60 9.
|
Artikolu 60 Ittestjar ta' sistemi tal-IA b'riskju għoli f'kundizzjonijiet tad-dinja reali barra mill-ambjenti ta' esperimentazzjoni regolatorja għall-IA
-
1. Ħarsa ġenerali
| Titlu |
Regolamentazzjoni |
|
Artikolu 60 1.
|
L-ittestjar tas-sistemi tal-IA b’riskju għoli f’kundizzjonijiet tad-dinja reali barra minn ambjenti ta’ esperimentazzjoni regolatorja għall-IA jista’ jitwettaq minn fornituri jew fornituri prospettivi tas-sistemi tal-IA b’riskju għoli elenkati fl-Anness III jew koperti mil-leġiżlazzjoni tal-Unjoni dwar l-armonizzazzjoni elenkata fit-Taqsima A tal-Anness I, f’konformità ma’ dan l-Artikolu u mal-pjan ta’ ttestjar fid-dinja reali msemmi f’dan l-Artikolu, mingħajr preġudizzju għall-projbizzjonijiet skont l-Artikolu 5. Il-Kummissjoni għandha, permezz ta’ atti ta’ implimentazzjoni, tispeċifika l-elementi dettaljati tal-pjan ta' ttestjar fid-dinja reali. Dawk l-atti ta’ implimentazzjoni għandhom jiġu adottati f'konformità mal-proċedura ta' eżami msemmija fl-Artikolu 98(2). Dan il-paragrafu għandu jkun mingħajr preġudizzju għal-leġiżlazzjoni tal-Unjoni jew dik nazzjonali dwar l-ittestjar fil-kundizzjonijiet tad-dinja reali ta' sistemi tal-IA b'riskju għoli relatati mal-prodotti koperti mil-leġiżlazzjoni tal-Unjoni dwar l-armonizzazzjoni elenkata fl-Anness I.
|
|
Artikolu 60 2.
|
Il-fornituri jew il-fornituri prospettivi jistgħu jwettqu l-ittestjar tas-sistemi tal-IA b’riskju għoli msemmija fl-Anness III jew koperti mil-leġiżlazzjoni tal-Unjoni dwar l-armonizzazzjoni elenkata fit-Taqsima A tal-Anness I f’kundizzjonijiet tad-dinja reali fi kwalunkwe ħin qabel l-introduzzjoni fis-suq jew it-tqegħid fis-servizz tas-sistema tal-IA b’riskju għoli, waħedhom jew bi sħubija ma’ implimentatur jew implimentatur prospettiv wieħed jew aktar.
|
|
Artikolu 60 3.
|
L-ittestjar ta' sistemi tal-IA b'riskju għoli f'kundizzjonijiet tad-dinja reali skont dan l-Artikolu għandu jkun mingħajr preġudizzju għal kwalunkwe rieżami etiku meħtieġ mil-liġi tal-Unjoni jew dik nazzjonali.
|
|
Artikolu 60 4.
|
Il-fornituri jew il-fornituri prospettivi jistgħu jwettqu l-ittestjar fil-kundizzjonijiet tad-dinja reali biss meta l-kundizzjonijiet kollha li ġejjin jiġu ssodisfati:
(a)
il-fornitur jew il-fornitur prospettiv ikun fassal pjan ta' ttestjar fid-dinja reali u ppreżentah lill-awtorità tas-sorveljanza tas-suq fl-Istat Membru fejn ser jitwettaq l-ittestjar fil-kundizzjonijiet tad-dinja reali;
(b)
l-awtorità tas-sorveljanza tas-suq fl-Istat Membru fejn għandu jitwettaq l-ittestjar fil-kundizzjonijiet tad-dinja reali tkun approvat l-ittestjar fil-kundizzjonijiet tad-dinja reali u l-pjan ta' ttestjar fid-dinja reali; meta l-awtorità tas-sorveljanza tas-suq ma tkunx ipprovdiet tweġiba fi żmien 30 jum, l-ittestjar fil-kundizzjonijiet tad-dinja reali u l-pjan ta' ttestjar fid-dinja reali għandhom jitqiesu bħala li ġew approvati; fejn il-liġi nazzjonali ma tipprevedix approvazzjoni taċita, l-ittestjar fil-kundizzjonijiet tad-dinja reali għandu jibqa' soġġett għal awtorizzazzjoni;
(c)
il-fornitur jew il-fornitur prospettiv, bl-eċċezzjoni ta' fornituri jew fornitur prospettivi tas-sistemi tal-IA b'riskju għoli msemmija fil-punti 1, 6 u 7 tal-Anness III fl-oqsma tal-infurzar tal-liġi, il-migrazzjoni, l-ażil u l-ġestjoni tal-kontrolli fil-fruntieri, u s-sistemi tal-IA b'riskju għoli msemmija fil-punt 2 tal-Anness III jkun irreġistra l-ittestjar fil-kundizzjonijiet tad-dinja reali f’konformità mal-Artikolu 71(4) b'numru uniku ta' identifikazzjoni waħdieni għall-Unjoni kollha u bl-informazzjoni speċifikata fl-Anness IX; il-fornitur jew il-fornitur prospettiv tas-sistemi tal-IA b'riskju għoli msemmija fil-punti 1, 6 u 7 tal-Anness III fl-oqsma tal-infurzar tal-liġi, il-migrazzjoni, l-ażil u l-ġestjoni tal-kontrolli fil-fruntieri, ikun irreġistra l-ittestjar fil-kundizzjonijiet tad-dinja reali fit-taqsima mhux pubblika sikura tal-bażi tad-data tal-UE skont l-Artikolu 49(4), il-punt (d), b’numru uniku ta’ identifikazzjoni waħdieni għall-Unjoni kollha u bl-informazzjoni speċifikata fih; il-fornitur jew il-fornitur prospettiv tas-sistemi tal-IA b'riskju għoli msemmija fil-punt 2 tal-Anness III ikun irreġistra l-ittestjar fil-kundizzjonijiet tad-dinja reali f’konformità tal-Artikolu 49(5);
(d)
il-fornitur jew il-fornitur prospettiv li jwettaq l-ittestjar fil-kundizzjonijiet tad-dinja reali jkun stabbilit fl-Unjoni jew ikun ħatar rappreżentant legali li jkun stabbilit fl-Unjoni;
(e)
id-data miġbura u pproċessata għall-fini tal-ittestjar fil-kundizzjonijiet tad-dinja reali għandha tiġi ttrasferita lejn pajjiżi terzi biss dment li jkunu implimentati salvagwardji xierqa u applikabbli skont il-liġi tal-Unjoni;
(f)
l-ittestjar fil-kundizzjonijiet tad-dinja reali ma jdumx aktar milli meħtieġ biex jintlaħqu l-objettivi tiegħu u fi kwalunkwe każ mhux aktar minn sitt xhur, li jistgħu jiġu estiżi għal perjodu ta' sitt xhur oħra, soġġett għal notifika minn qabel mill-fornitur jew mill-fornitur prospettiv lill-awtorità tas-sorveljanza tas-suq, akkumpanjat minn spjegazzjoni dwar il-ħtieġa għal tali estensjoni;
(g)
is-suġġetti tal-ittestjar fil-kundizzjonijiet tad-dinja reali li huma persuni li jagħmlu parti minn gruppi vulnerabbli minħabba l-età, id-diżabbiltà tagħhom, huma protetti kif xieraq;
(h)
meta fornitur jew fornitur prospettiv jorganizza l-ittestjar fil-kundizzjonijiet tad-dinja reali f'kooperazzjoni ma' implimentatur jew implimentatur prospettiv wieħed jew aktar, dawn tal-aħħar ikunu ġew infurmati dwar l-aspetti kollha tal-ittestjar li huma rilevanti għad-deċiżjoni tagħhom li jipparteċipaw, u mogħtija l-istruzzjonijiet rilevanti dwar l-użu tas-sistema tal-IA msemmija fl-Artikolu 13; il-fornitur jew il-fornitur prospettiv u l-implimentatur jew l-implimentatur prospettiv għandhom jikkonkludu ftehim li jispeċifika r-rwoli u r-responsabbiltajiet tagħhom bil-ħsieb li tiġi żgurata l-konformità mad-dispożizzjonijiet għall-ittestjar fil-kundizzjonijiet tad-dinja reali skont dan ir-Regolament u skont liġijiet applikabbli oħra tal-Unjoni u nazzjonali;
(i)
is-suġġetti tal-ittestjar fil-kundizzjonijiet tad-dinja reali taw kunsens infurmat f'konformità mal-Artikolu 61, jew fil-każ tal-infurzar tal-liġi, meta t-tiftix ta' kunsens infurmat jipprevjeni li s-sistema tal-IA tiġi ttestjata, l-ittestjar innifsu u l-eżitu tal-ittestjar fil-kundizzjonijiet tad-dinja reali ma għandu jkollhom ebda effett negattiv fuq is-suġġetti u d-data personali tagħhom għandha titħassar wara li jkun twettaq it-test;
(j)
l-ittestjar fil-kundizzjonijiet tad-dinja reali huwa ssorveljat b'mod effettiv mill-fornitur jew mill-fornitur prospettiv, kif ukoll mill-implimentaturi jew l-implimentaturi prospettivi permezz ta' persuni li huma kkwalifikati kif xieraq fil-qasam rilevanti u li għandhom il-kapaċità, it-taħriġ u l-awtorità meħtieġa biex iwettqu l-kompiti tagħhom;
(k)
il-previżjonijiet, ir-rakkomandazzjonijiet jew id-deċiżjonijiet tas-sistema tal-IA jistgħu jitreġġgħu lura u jiġu injorati b'mod effettiv.
|
|
Artikolu 60 5.
|
Is-suġġetti tal-ittestjar fil-kundizzjonijiet tad-dinja reali, jew ir-rappreżentant iddeżinjat legalment tagħhom, kif xieraq, jistgħu, mingħajr ebda detriment li jirriżulta u mingħajr ma jkollhom għalfejn jipprovdu xi ġustifikazzjoni, jirtiraw mill-ittestjar fi kwalunkwe ħin billi jirrevokaw il-kunsens infurmat tagħhom u jistgħu jitolbu t-tħassir immedjat u permanenti tad-data personali tagħhom. L-irtirar tal-kunsens infurmat ma għandux jaffettwa l-attivitajiet diġà mwettqa.
|
|
Artikolu 60 6.
|
F'konformità mal-Artikolu 75, l-Istati Membri għandhom jagħtu lill-awtoritajiet tas-sorveljanza tas-suq tagħhom is-setgħat li jirrikjedu lill-fornituri u lill-fornituri prospettivi biex jipprovdu informazzjoni, li jwettqu spezzjonijiet mill-bogħod jew fuq il-post għal għarrieda u li jwettqu kontrolli fuq it-twettiq tal-ittestjar fil-kundizzjonijiet tad-dinja reali u s-sistemi tal-IA b’riskju għoli relatati. L-awtoritajiet tas-sorveljanza tas-suq għandhom jużaw dawk is-setgħat biex jiżguraw l-iżvilupp sikur tal-ittestjar fil-kundizzjonijiet tad-dinja reali.
|
|
Artikolu 60 7.
|
Kwalunkwe inċident serju identifikat matul l-ittestjar fil-kundizzjonijiet tad-dinja reali għandu jiġi rrappurtat lill-awtorità nazzjonali tas-sorveljanza tas-suq skont l-Artikolu 73. Il-fornitur jew fornitur prospettiv għandu jadotta miżuri ta' mitigazzjoni immedjati jew, fin-nuqqas ta' dan, għandu jissospendi l-ittestjar fil-kundizzjonijiet tad-dinja reali sakemm isseħħ tali mitigazzjoni jew inkella jtemmu. Il-fornitur jew fornitur prospettiv għandu jistabbilixxi proċedura biex isejjaħ lura fil-pront is-sistema tal-IA mat-tali terminazzjoni tal-ittestjar fil-kundizzjonijiet tad-dinja reali.
|
|
Artikolu 60 8.
|
Il-fornituri jew il-fornituri prospettivi għandhom jinnotifikaw lill-awtorità nazzjonali tas-sorveljanza tas-suq fl-Istat Membru fejn l-ittestjar fil-kundizzjonijiet tad-dinja reali għandu jitwettaq bis-sospensjoni jew it-terminazzjoni tal-ittestjar fil-kundizzjonijiet tad-dinja reali u tal-eżiti finali.
|
|
Artikolu 60 9.
|
Il-fornitur jew il-fornitur prospettiv għandu jkun responsabbli skont il-liġijiet applikabbli tal-Unjoni u nazzjonali dwar ir-responsabbiltà għal kwalunkwe dannu kkawżat matul l-ittestjar fil-kundizzjonijiet tad-dinja reali.
|
,
1.1 Referenzi
1.2 Rekwiżiti identifikati
1.3 Standards relatati
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- AI TEVV Reporting (SCF)
- AI TEVV Reporting (SCF)
- Serious Incident Reporting For AI & Autonomous Technologies (SCF)
2. Rekwiżiti identifikati
3. Standards relatati
Standards
| Sors |
Rekwiżit |
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program
∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security)
∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program
∙ Formal AI TEVV process for AI systems
∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program
∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function
∙ AI testing tools (e.g., IBM OpenScale, Great Expectations)
∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program
∙ NIST AI RMF Measure function
∙ Third-party AI testing for high-risk systems
∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program
∙ Enterprise AI TEVV program (NIST AI RMF Measure function)
∙ Independent AI testing and evaluation for high-risk systems
∙ AI TEVV integrated with MLOps and CI/CD
∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
AI TEVV Reporting
Description
Mechanisms exist to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report AI testing results to management
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Document and communicate AI test results to stakeholders
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal AI TEVV reporting process
∙ Standardized reporting templates
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI TEVV reporting program
∙ Regular stakeholder briefings
∙ Findings tracking
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI TEVV reporting framework
∙ Automated test result dashboards
∙ Board-level AI risk reporting
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
AI TEVV Reporting
Description
Mechanisms exist to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report AI testing results to management
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Document and communicate AI test results to stakeholders
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal AI TEVV reporting process
∙ Standardized reporting templates
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI TEVV reporting program
∙ Regular stakeholder briefings
∙ Findings tracking
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI TEVV reporting framework
∙ Automated test result dashboards
∙ Board-level AI risk reporting
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control:
▪ A reasonable person would conclude this control requires a structured process.
▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control:
▪ A reasonable person would conclude a well-defined and standardized process is required.
▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization.
▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Serious Incident Reporting For AI & Autonomous Technologies
Description
Mechanisms exist to report any serious incident involving operational Artificial Intelligence (AI) and Autonomous Technologies (AAT) to relevant authorities as to when and where the serious incident occurred, in accordance with mandated reporting timelines.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report serious AI incidents to relevant authorities as required
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Incident reporting procedure for serious AI incidents
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal serious AI incident reporting process
∙ Regulatory reporting contacts
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI incident response plan with regulatory reporting requirements
∙ Legal team involvement
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI incident response program
∙ Automated regulatory notification workflows
∙ Legal and compliance integration
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
Artificial Intelligence and Autonomous Technology (AAT) domain capabilities are ad hoc and inconsistent. Capability criteria associated with this control may include:
▪ Policies, standards & procedures associated with AAT domain capabilities provide limited coverage due to the depth and breadth of the existing documentation.
▪ AAT-related processes are expected to follow the organization's existing processes (e.g., incident response, asset management, change control and risk assessments).
▪ No formal Governance, Risk & Compliance (GRC) team exists to provide AAT oversight, where the Chief Information Officer (CIO), or similar function, governs technology decisions what is acceptable for AAT within the organization.
Level 2 Planned Tracked
Artificial Intelligence and Autonomous Technology (AAT) capabilities are requirements-driven, but are not standardized across the entity (e.g., local/regional level consistency). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are documented and maintained by process owners.
▪ IT and/or cybersecurity personnel work with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address applicable statutory, regulatory and/or contractual requirements for Technology Assets, Applications, Services and/or Data (TAASD).
▪ Artificial Intelligence (AI)-related controls are primarily administrative and preventative in nature (e.g., policies, standards and procedures & guidelines).
▪ Asset management may be a defined function (e.g., team or department) or assigned as an additional duty to existing IT and/or cybersecurity personnel.
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist:
▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function.
▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners.
▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities.
▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform).
▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls).
▪ An implemented and operational capability exists to report any serious incident involving operational AAT to relevant authorities as to when and where the serious incident occurred, in accordance with mandated reporting timelines.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions:
▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control.
▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions:
▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control.
▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies.
▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|