+Článek 60 Testování vysoce rizikových systémů AI v reálných podmínkách mimo regulační sandboxy pro AI
---+Článek 60 1.
---+Článek 60 2.
---+Článek 60 3.
---+Článek 60 4.
---+Článek 60 5.
---+Článek 60 6.
---+Článek 60 7.
---+Článek 60 8.
---+Článek 60 9.
|
Článek 60 Testování vysoce rizikových systémů AI v reálných podmínkách mimo regulační sandboxy pro AI
-
1. Přehled
| Název |
Regulace |
|
Článek 60 1.
|
Poskytovatelé nebo potenciální poskytovatelé vysoce rizikových systémů AI uvedených v příloze III, nebo vysoce rizikových systémů AI, na které se vztahují harmonizační právní předpisy Unie uvedené v příloze I oddíle A, mohou provádět testování vysoce rizikových systémů AI v reálných podmínkách mimo regulační sandboxy pro AI v souladu s tímto článkem a plánem testování v reálných podmínkách uvedeným v tomto článku, aniž jsou dotčeny zákazy podle článku 5. Komise v prováděcích aktech upřesní podrobné prvky plánu testování v reálných podmínkách. Tyti prováděcí akty budou přijaty přezkumným postupem podle čl. 98 odst. 2. Tímto odstavcem není dotčeno právo Unie ani vnitrostátní právo týkající se testování vysoce rizikových systémů AI v reálných podmínkách v souvislosti s produkty, na něž se vztahují harmonizační právní předpisy uvedené v příloze I.
|
|
Článek 60 2.
|
Poskytovatelé nebo potenciální poskytovatelé mohou provádět testování vysoce rizikových systémů AI uvedených v příloze III, nebo vysoce rizikových systémů AI, na které se vztahují harmonizační právní předpisy Unie uvedené v příloze I oddíle A, v reálných podmínkách kdykoli před uvedením vysoce rizikového systému AI na trh nebo do provozu, a to samostatně nebo v partnerství s jedním nebo více zavádějícími subjekty nebo potenciálními zavádějícími subjekty.
|
|
Článek 60 3.
|
Testováním vysoce rizikových systémů AI v reálných podmínkách podle tohoto článku není dotčen žádný etický přezkum, který je vyžadován právem Unie nebo vnitrostátním právem.
|
|
Článek 60 4.
|
Poskytovatelé nebo potenciální poskytovatelé mohou provádět testování v reálných podmínkách pouze v případě, že jsou splněny všechny tyto podmínky:
a)
poskytovatel nebo potenciální poskytovatel vypracoval plán testování v reálných podmínkách a předložil jej orgánu dozoru nad trhem v členském státě, v němž má být testování v reálných podmínkách provedeno;
b)
vnitrostátní orgán dozoru nad trhem v členském státě, v němž se testování v reálných podmínkách má provést, testování v reálných podmínkách a plán testování v reálných podmínkách schválil; pokud orgán dozoru nad trhem neposkytne odpověď do 30 dnů, testování v reálných podmínkách a plán zkoušek v reálných podmínkách se považují za schválené; nestanoví-li vnitrostátní právo implicitní schválení, testování v reálných podmínkách nadále podléhá povolení;
c)
poskytovatel nebo potenciální poskytovatel, s výjimkou poskytovatelů nebo potenciálních poskytovatelů vysoce rizikových systémů AI uvedených v bodech 1, 6 a 7 přílohy III v oblasti vymáhání práva, migrace, azylu a řízení ochrany hranic a vysoce rizikových systémů AI uvedených v bodě 2 přílohy III, zaregistroval testování v reálných podmínkách v souladu s čl. 71 odst. 4 s uvedením celounijního jedinečného identifikačního čísla a informací stanovených v příloze IX; poskytovatel nebo potenciální poskytovatel vysoce rizikových systémů AI uvedených v bodech 1, 6 a 7 přílohy III v oblasti vymáhání práva, migrace, azylu a řízení ochrany hranic zaregistroval testování v reálných podmínkách v zabezpečené, neveřejné části databáze EU podle čl. 49 odst. 4 písm. d) s celounijním jedinečným identifikačním číslem a informacemi v tomto článku uvedenými; poskytovatel nebo potenciální poskytovatel vysoce rizikových systémů AI uvedený v bodě 2 přílohy III zaregistroval testování v reálných podmínkách v souladu s čl. 49 odst. 5;
d)
poskytovatel nebo potenciální poskytovatel provádějící testování v reálných podmínkách je usazen v Unii nebo jmenoval právního zástupce, který je usazen v Unii;
e)
údaje shromážděné a zpracované pro účely testování v reálných podmínkách se předávají třetím zemím pouze za předpokladu, že jsou uplatňovány vhodné a použitelné záruky podle práva Unie;
f)
testování v reálných podmínkách netrvá déle, než je nezbytné k dosažení jeho cílů, a v žádném případě ne déle než šest měsíců, přičemž tato doba může být prodloužena o dalších šest měsíců, pokud poskytovatel nebo potenciální poskytovatel toto prodloužení předem oznámí orgánu dozoru nad trhem spolu s vysvětlením, proč je ho zapotřebí;
g)
subjekty testování v reálných podmínkách, které vzhledem ke svému věku nebo postižení spadají do zranitelných skupin, jsou náležitě chráněny;
h)
pokud poskytovatel nebo potenciální poskytovatel uspořádá testování v reálných podmínkách ve spolupráci s jedním nebo více zavádějícími subjekty či potenciálními zavádějícími subjekty, tyto subjekty byly informovány o všech aspektech testování, které jsou relevantní pro jejich rozhodnutí účastnit se, a obdrželi příslušné pokyny pro použití systému AI uvedeného v článku 13; poskytovatel nebo potenciální poskytovatel a zavádějící subjekt nebo potenciální zavádějící subjekt uzavřou dohodu, v níž upřesní své úlohy a povinnosti s cílem zajistit soulad s ustanoveními týkajícími se testování v reálných podmínkách podle tohoto nařízení a jiným platným právem Unie a vnitrostátním právem;
i)
subjekty testování v reálných podmínkách udělily informovaný souhlas v souladu s článkem 61, nebo pokud by v případě vymáhání práva získání informovaného souhlasu testování systému AI bránilo, samotné testování a výsledek testování v reálných podmínkách nemají na subjekty hodnocení žádný negativní dopad a osobní údaje těchto subjektů jsou po provedení testu vymazány;
j)
na testování v reálných podmínkách účinně dohlíží poskytovatel nebo potenciální poskytovatel, jakož i zavádějící subjekty nebo potenciální zavádějící subjekty prostřednictvím osob, které mají jak odpovídající kvalifikaci v příslušné oblasti, tak nezbytnou kapacitu, odbornou přípravu a pravomoc k plnění svých úkolů;
k)
predikce, doporučení nebo rozhodnutí systému AI lze účinně zvrátit a nezohlednit.
|
|
Článek 60 5.
|
Kterékoli subjekty testování v reálných podmínkách nebo případně jejich zákonně ustanovený zástupce může odvoláním svého informovaného souhlasu od testování kdykoli odstoupit, aniž by tím došel jakékoliv újmy a aniž by byl povinen poskytnout jakékoliv odůvodnění, a může požádat o okamžité a trvalé vymazání svých osobních údajů. Odvoláním informovaného souhlasu nejsou dotčeny již provedené činnosti.
|
|
Článek 60 6.
|
V souladu s článkem 75 svěří členské státy svým orgánům dozoru nad trhem pravomoci požadovat od poskytovatelů a potenciálních poskytovatelů informace, provádět neohlášené kontroly na dálku nebo kontroly na místě a provádět kontroly postupu testování v reálných podmínkách a souvisejících vysoce rizikových systémů AI. Orgány dozoru nad trhem využijí těchto pravomocí k zajištění bezpečného vývoje testování v reálných podmínkách.
|
|
Článek 60 7.
|
Veškeré závažné incidenty zjištěné v průběhu testování v reálných podmínkách se oznámí vnitrostátnímu orgánu dozoru nad trhem v souladu s článkem 73. Poskytovatel nebo potenciální poskytovatel přijme okamžitá zmírňující opatření nebo, pokud to není možné, testování v reálných podmínkách pozastaví, dokud k tomuto zmírnění nedojde, nebo testování jiným způsobem ukončí. Po takovém ukončení testování v reálných podmínkách stanoví poskytovatel nebo potenciální poskytovatel postup pro okamžité stažení systému AI z oběhu.
|
|
Článek 60 8.
|
Poskytovatelé nebo potenciální poskytovatelé oznámí vnitrostátnímu orgánu dozoru nad trhem v členském státě, v němž se testování v reálných podmínkách provádí, že testování v reálných podmínkách bylo pozastaveno nebo ukončeno, a jaké jsou konečné výsledky.
|
|
Článek 60 9.
|
Podle platného práva Unie a vnitrostátního práva upravujícího odpovědnost nesou poskytovatel nebo potenciální poskytovatel odpovědnost za veškeré škody, jež byly v průběhu jejich testování v reálných podmínkách způsobeny.
|
,
1.1 Odkazy
1.2 Identifikované požadavky
1.3 Související standardy
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) (SCF)
- AI TEVV Reporting (SCF)
- AI TEVV Reporting (SCF)
- Serious Incident Reporting For AI & Autonomous Technologies (SCF)
2. Identifikované požadavky
Požadavky
| Zdroj |
Požadavek |
3. Související standardy
Standardy
| Zdroj |
Požadavek |
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV)
Description
Mechanisms exist to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable Artificial Intelligence (AI) and Autonomous Technologies (AAT)-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Information Assurance (IA) Program ∙ AI TEVV checklist for AI tools (test accuracy, validate outputs, verify security) ∙ AI governance program
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Information Assurance (IA) Program ∙ Formal AI TEVV process for AI systems ∙ AI governance program
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Information Assurance (IA) Program ∙ Formal AI TEVV framework aligned to NIST AI RMF Measure function ∙ AI testing tools (e.g., IBM OpenScale, Great Expectations) ∙ AI governance program
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program ∙ NIST AI RMF Measure function ∙ Third-party AI testing for high-risk systems ∙ AI testing integrated with CI/CD pipelines
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Information Assurance (IA) Program ∙ Enterprise AI TEVV program (NIST AI RMF Measure function) ∙ Independent AI testing and evaluation for high-risk systems ∙ AI TEVV integrated with MLOps and CI/CD ∙ EU AI Act conformity assessment (if applicable)
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to implement Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) practices to enable AAT-related security, resilience and compliance-related conformity testing throughout the lifecycle of the AAT.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
AI TEVV Reporting
Description
Mechanisms exist to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report AI testing results to management
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Document and communicate AI test results to stakeholders
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal AI TEVV reporting process ∙ Standardized reporting templates
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI TEVV reporting program ∙ Regular stakeholder briefings ∙ Findings tracking
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI TEVV reporting framework ∙ Automated test result dashboards ∙ Board-level AI risk reporting
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
AI TEVV Reporting
Description
Mechanisms exist to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report AI testing results to management
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Document and communicate AI test results to stakeholders
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal AI TEVV reporting process ∙ Standardized reporting templates
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI TEVV reporting program ∙ Regular stakeholder briefings ∙ Findings tracking
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI TEVV reporting framework ∙ Automated test result dashboards ∙ Board-level AI risk reporting
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
SCR-CMM Level 1 criteria definitions are not available for this control: ▪ A reasonable person would conclude this control requires a structured process. ▪ At this level of maturity, the "ad hoc" nature of performing a capability informally would indicate the intent of the control is not met due to a lack of consistency and formality.
Level 2 Planned Tracked
SCR-CMM Level 2 criteria definitions are not available for this control: ▪ A reasonable person would conclude a well-defined and standardized process is required. ▪ At this level of maturity, the “requirements-driven” nature of performing the control is focused on a localized and/or regionalized implementation, not uniform and consistent across the organization. ▪ Requirements are narrowly scoped for applicability and are primarily derived from compliance obligations (e.g., laws, regulations and contracts).
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to report the status and results of Artificial Intelligence Test, Evaluation, Validation & Verification (AI TEVV) to relevant stakeholders, including governing bodies, as required.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|
SCF
|
Serious Incident Reporting For AI & Autonomous Technologies
Description
Mechanisms exist to report any serious incident involving operational Artificial Intelligence (AI) and Autonomous Technologies (AAT) to relevant authorities as to when and where the serious incident occurred, in accordance with mandated reporting timelines.
Possible Solutions & Considerations
Micro-Small Business (<10 staff) / BLS Firm Size Classes 1-2
∙ Report serious AI incidents to relevant authorities as required
Small Business (10-49 staff) / BLS Firm Size Classes 3-4
∙ Incident reporting procedure for serious AI incidents
Medium Business (50-249 staff) / BLS Firm Size Classes 5-6
∙ Formal serious AI incident reporting process ∙ Regulatory reporting contacts
Large Business (250-999 staff) / BLS Firm Size Classes 7-8
∙ AI incident response plan with regulatory reporting requirements ∙ Legal team involvement
Enterprise (> 1,000 staff) / BLS Firm Size Class 9
∙ Enterprise AI incident response program ∙ Automated regulatory notification workflows ∙ Legal and compliance integration
SCR-CMM
Level 0 Not Performed
Practices are non-existent, based on the inability to demonstrate an implemented and operational capability. A reasonable person would conclude the control is not being performed.
Level 1 Performed Informally
Artificial Intelligence and Autonomous Technology (AAT) domain capabilities are ad hoc and inconsistent. Capability criteria associated with this control may include: ▪ Policies, standards & procedures associated with AAT domain capabilities provide limited coverage due to the depth and breadth of the existing documentation. ▪ AAT-related processes are expected to follow the organization's existing processes (e.g., incident response, asset management, change control and risk assessments). ▪ No formal Governance, Risk & Compliance (GRC) team exists to provide AAT oversight, where the Chief Information Officer (CIO), or similar function, governs technology decisions what is acceptable for AAT within the organization.
Level 2 Planned Tracked
Artificial Intelligence and Autonomous Technology (AAT) capabilities are requirements-driven, but are not standardized across the entity (e.g., local/regional level consistency). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are documented and maintained by process owners. ▪ IT and/or cybersecurity personnel work with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address applicable statutory, regulatory and/or contractual requirements for Technology Assets, Applications, Services and/or Data (TAASD). ▪ Artificial Intelligence (AI)-related controls are primarily administrative and preventative in nature (e.g., policies, standards and procedures & guidelines). ▪ Asset management may be a defined function (e.g., team or department) or assigned as an additional duty to existing IT and/or cybersecurity personnel.
Level 3 Well Defined
Artificial Intelligence and Autonomous Technology (AAT) capabilities are standardized across the entity for applicability to People, Processes, Technologies, Data and/or Facilities (PPTDF) to ensure consistency for Technology Assets, Applications, Services and/or Data (TAASD). Capability criteria associated with this control reasonably expect the following criteria to exist: ▪ Policies and standards associated with AAT domain capabilities are formally documented and centrally-managed by the entity's Governance, Risk & Compliance (GRC) team, or similar function. ▪ Standardized Operating Procedures (SOP) associated with AAT domain capabilities are well-documented and kept current by process owners. ▪ An Artificial Intelligence Governance (AIG) team, or similar function, is appropriately staffed and supported to implement and maintain AAT domain capabilities. ▪ Technology is leveraged to enhance the efficiency and accuracy of AI governance, risk management and compliance operations (e.g., dedicated AI governance platform). ▪ The entity's Governance, Risk & Compliance (GRC) team, or similar function, works with business stakeholders and process owners to appropriately scope and reasonably implement cybersecurity and data protection controls associated with AAT domain capabilities to address Minimum Compliance Requirements (MCR) (e.g., applicable statutory, regulatory and/or contractual requirements) and Discretionary Security Requirements (DSR) (e.g., entity-required controls). ▪ An implemented and operational capability exists to report any serious incident involving operational AAT to relevant authorities as to when and where the serious incident occurred, in accordance with mandated reporting timelines.
Level 4 Quantitatively Controlled
Utilize SCR-CMM Level 3 criteria definitions: ▪ There are no defined Level 4 criteria, since it is reasonable to assume a quantitatively-controlled process is not necessary to operationalize this control. ▪ While it may be possible to develop “metrics-driven” capabilities for this control, the criteria would be organization-specific to define.
Level 5 Continuously Improving
Utilize SCR-CMM Level 3 or Level 4 (if available) criteria definitions: ▪ There are no defined Level 5 criteria, since it is reasonable to assume a continuously-improving process is not necessary to operationalize this control. ▪ Level 5 capabilities should be considered “world-class” where the control builds on Level 4 capabilities, but are continuously improving through Artificial Intelligence (AI) and/or Machine Learning (ML) technologies. ▪ While it may be possible to develop responsive capabilities for this control through the use of AI and/or ML technologies, the criteria would be organization-specific to define.
|
|