AI Risk and Opportunity Management
There is an old saying in the world of finance — attributed, with perhaps too much certainty, to various luminaries — that risk and return are inseparable companions. You cannot reach for one without accepting the other. Artificial intelligence, in the enterprise context, exemplifies this truth with unusual clarity. The same properties that make AI systems powerful — their capacity to learn from vast data, to identify patterns invisible to human perception, to act at machine speed — are precisely the properties that make them risky. An AI system that learns from biased data learns bias at scale. One that acts at machine speed can propagate an error at machine speed. One that identifies patterns no human has explicitly programmed can identify patterns that no human intended to surface.
Domain 2 of the AAISM examination confronts this tension directly. It is the domain of AI risk and opportunity management — concerned not with the elimination of risk, which is neither possible nor desirable, but with its disciplined identification, assessment, classification, treatment, and monitoring. It is also, importantly, concerned with opportunity — the recognition that managing AI risk well is itself a source of competitive advantage, regulatory resilience, and stakeholder trust.
This chapter is structured around three sub-domains: the frameworks, assessments, and treatment strategies that constitute Part A; the threat and vulnerability landscape of AI systems that constitutes Part B; and the vendor and supply chain management disciplines that constitute Part C. Together, these three areas cover the full thirty-one percent of the AAISM examination assigned to Domain 2.
- Domain 2 covers three sub-areas: (2A) AI Risk Assessment, Thresholds, and Treatment; (2B) AI Threat and Vulnerability Management; (2C) AI Vendor and Supply Chain Management.
- AI risk is qualitatively different from traditional IT risk: it is probabilistic, adaptive, often opaque, and involves novel attack surfaces that conventional security frameworks do not fully address.
- Risk and opportunity are inseparable in AI — the goal is not to minimise all risk but to accept risk intelligently, at levels commensurate with expected value, and with controls proportionate to impact.
- The NIST AI RMF (Govern, Map, Measure, Manage) and the EU AI Act's risk tier classification are the two most frequently examined frameworks in this domain.
AI Risk Assessment, Thresholds, and Treatment
2.1 The Nature of AI Trust
Before an enterprise can manage AI risk, it must grapple with a prior question: to what extent can it trust its AI systems? Trust, in the AI context, is not an intuitive feeling or a marketing claim. It is an earned, evidence-based judgement that a particular AI system will behave reliably, safely, fairly, and within its intended boundaries across a range of conditions — including conditions that were not explicitly anticipated during design and testing.
AI trust is multi-dimensional. A model may be trusted to be accurate — in the sense of performing well on standard benchmarks — while remaining untrustworthy with respect to fairness (performing differently across demographic groups), robustness (failing under adversarial inputs), or transparency (being unable to explain its outputs in ways that allow meaningful human oversight). Governance of AI trust requires attention to all these dimensions simultaneously.
The concept of trust is also relational and contextual. An AI model that is appropriately trusted for one application may be dangerously over-trusted in another. A language model that produces reliable summaries of internal documents may hallucinate convincingly when asked for medical or legal guidance. The enterprise's risk management function must continuously calibrate the level of trust extended to each AI system against the specific context in which it operates, the criticality of the decisions it influences, and the consequences of its failures.
- The Quranic injunction to verify information before acting — 'O you who believe, if a wrongdoer brings you some news, investigate' (49:6) — captures something essential about AI trust. The speed and confidence with which AI systems produce outputs creates powerful pressure to accept them uncritically. Effective AI risk management requires us to resist that pressure and to establish verification structures proportionate to the consequences of being wrong.
- Consider: Your organisation deploys a generative AI tool that staff trust because it is rarely obviously wrong. But rare and obvious errors are not the primary risk of sophisticated AI — subtle, systematic, hard-to-detect errors are. How would your governance framework surface errors that are neither rare nor obvious?
- Trust, once extended inappropriately to an AI system, is extremely difficult to withdraw. The governance discipline of establishing trust criteria before deployment, rather than after a failure, is far more effective and far less costly.
2.2 AI Risk Identification
Risk identification is the foundational activity of AI risk management. Before risks can be assessed, treated, or monitored, they must first be discovered — and the AI risk landscape is significantly broader and more varied than the risk landscapes of conventional IT systems.
AI risks can be categorised along several dimensions. By source: they may arise from the data used to train and operate AI systems, from the model architecture and training process, from the deployment environment and integration with other systems, from the behaviour of users and other human actors, or from external actors attempting to exploit AI vulnerabilities. By nature: they may be technical (the model fails to perform as intended), ethical (the model performs as intended but produces unfair or harmful outcomes), regulatory (the model's operation violates applicable law), or reputational (the model's behaviour, when disclosed, damages public trust). By timing: risks may materialise during development (poorly chosen training data), deployment (inadequate access controls), operation (model drift), or retirement (improper disposal of training data or model artefacts).
An effective AI risk identification process is not a one-time exercise. AI systems are dynamic — their performance evolves as real-world conditions change, as users discover new ways to interact with them, and as adversaries develop new techniques for exploiting them. Risk identification must therefore be embedded in the ongoing governance and monitoring of AI systems, not confined to a pre-deployment assessment.
2.3 AI Risk Frameworks: NIST AI RMF and the EU AI Act
Two frameworks dominate the AI risk management landscape and appear with particular frequency in AAISM examination questions: the NIST Artificial Intelligence Risk Management Framework and the European Union AI Act. They approach AI risk from different angles — one as a voluntary technical framework, the other as binding law — but they are complementary, and the AAISM professional must understand both.
2.3.1 The NIST Artificial Intelligence Risk Management Framework
The NIST AI Risk Management Framework — commonly abbreviated as the NIST AI RMF — was developed by the United States National Institute of Standards and Technology and published in January 2023. It is a voluntary, consensus-based framework designed to help organisations manage AI risks across the full AI life cycle. It does not impose mandatory requirements but provides a structured, practical approach that organisations of any size, sector, or jurisdiction can adapt to their specific context.
The NIST AI RMF is organised around four core functions that together constitute a continuous risk management cycle. These four functions — Govern, Map, Measure, and Manage — are deliberately sequenced to reflect the operational logic of mature risk management.
| Function | Description |
|---|---|
| GOVERN | Establishes the organisational culture, structures, policies, and processes needed to enable effective AI risk management. Governance activities include establishing risk management roles and responsibilities, defining risk tolerance, creating policies for AI development and deployment, and ensuring accountability. GOVERN is the enabling function — without it, the other three functions lack direction and authority. |
| MAP | Identifies and categorises AI risks in context. Mapping activities include characterising the AI system and its deployment context, identifying relevant stakeholders and their interests, surfacing potential harms and benefits, and categorising risks by type, likelihood, and potential impact. MAP translates the abstract concept of AI risk into a concrete, system-specific risk register. |
| MEASURE | Analyses and assesses identified AI risks, using both qualitative and quantitative methods. Measurement activities include testing and evaluating AI system performance, assessing the likelihood and impact of identified risk scenarios, evaluating existing controls for adequacy, and tracking risk-relevant metrics over time. MEASURE provides the evidentiary basis for informed risk treatment decisions. |
| MANAGE | Prioritises, responds to, and monitors AI risks. Management activities include selecting and implementing risk treatments, allocating resources to risk reduction and response, monitoring AI system performance and risk indicators on an ongoing basis, and adapting risk management practices as conditions change. MANAGE is where risk identification and assessment translate into organisational action. |
The NIST AI RMF is notable for its emphasis on trustworthiness as a central organising principle. It identifies seven properties of trustworthy AI: accountability, explainability, fairness, interpretability, privacy-enhanced, reliable, and safe. These properties are not merely aspirational — they are the criteria against which AI risk management activities should be evaluated.
2.3.2 The European Union AI Act — A Deeper Look
The EU AI Act, which entered into force in August 2024 with a phased implementation schedule, is the world's first comprehensive, binding AI regulation. Its risk-based architecture classifies AI systems into four tiers according to the level of risk they pose to health, safety, and fundamental rights.
| Risk Tier | Description and Requirements |
|---|---|
| Unacceptable Risk (Prohibited) | AI systems that pose an unacceptable risk to fundamental rights or human dignity are prohibited outright. Examples include AI-powered social scoring by public authorities, real-time biometric surveillance in public spaces (with narrow exceptions), and AI that manipulates human behaviour through subliminal techniques. |
| High Risk | AI systems that pose significant risk to health, safety, or fundamental rights but are not prohibited. High-risk systems are subject to stringent requirements: conformity assessments, technical documentation, human oversight mechanisms, data governance requirements, transparency obligations, and post-market monitoring. High-risk categories include AI in critical infrastructure, education, employment, essential services, border control, and the administration of justice. |
| Limited Risk | AI systems that pose limited risk, primarily through transparency failures. These systems must comply with specific transparency obligations — for example, a chatbot must disclose that it is AI, and deepfake content must be labelled. The obligations are less onerous than for high-risk systems. |
| Minimal Risk | AI systems that pose minimal or negligible risk. These systems are not subject to specific mandatory requirements, though organisations are encouraged to adopt voluntary codes of conduct. The vast majority of AI systems currently in commercial use fall into this category. |
2.3.3 NIST AI RMF vs. EU AI Act: Key Differences
The two frameworks are complementary but differ in important respects that the AAISM professional must appreciate. The NIST AI RMF is voluntary and process-oriented — it tells organisations how to manage AI risk without prescribing specific outcomes. The EU AI Act is mandatory and outcome-oriented — it defines what high-risk AI systems must achieve and imposes legal sanctions for non-compliance. The NIST framework is applicable globally to any organisation that chooses to adopt it; the EU Act applies to all AI systems placed on or used in the EU market, regardless of the developer's location.
In practice, the two frameworks are mutually reinforcing. An organisation that has implemented the NIST AI RMF's Govern, Map, Measure, and Manage functions will be well-positioned to satisfy the EU AI Act's conformity assessment, documentation, and oversight requirements for high-risk systems. Conversely, the EU AI Act's specific requirements for high-risk systems provide concrete, actionable targets for the NIST framework's more general guidance.
- Know the four NIST AI RMF functions cold: GOVERN, MAP, MEASURE, MANAGE. Examination questions frequently test the correct sequencing of these functions and which activities belong to which function.
- The EU AI Act's four risk tiers — Unacceptable, High, Limited, Minimal — are frequently tested. Remember that 'high risk' does not mean prohibited — it means subject to stringent compliance requirements.
- When a question describes a scenario involving an AI system being deployed in a new context, the FIRST risk management activity is almost always risk identification and classification (MAP in the NIST framework) — not immediate control implementation.
2.4 AI Risk Classification, Impact Assessments, and Acceptable Limits
2.4.1 Risk Classification
Risk classification assigns AI risks to defined categories based on their nature, severity, and the governance response they require. A well-designed classification system enables the risk management function to prioritise attention, allocate resources proportionately, and communicate risk clearly to governance bodies that must make resource allocation decisions.
Common AI risk classification dimensions include: the probability of occurrence (how likely is this risk to materialise, given current controls?); the magnitude of impact if it does materialise (how much harm would result — financial, reputational, regulatory, human?); the speed of onset (would the risk materialise suddenly or gradually?); the reversibility of harm (could the damage be remediated, or would it be permanent?); and the breadth of impact (would harm be confined to the organisation, or would it affect customers, third parties, or the broader public?). Each of these dimensions contributes to an overall risk rating that guides the intensity of the governance response.
2.4.2 Fundamental Rights Impact Assessment for High-Risk AI Systems
The Fundamental Rights Impact Assessment (FRIA) is a structured evaluation process required by the EU AI Act for certain deployers of high-risk AI systems. It assesses the potential impact of an AI system on the fundamental rights of individuals and groups — including the right to privacy, the right to non-discrimination, the right to an effective remedy, and the right to human dignity.
Conducting a FRIA requires the AI security professional to go beyond technical risk assessment and engage with the broader social and human context of AI deployment. Who are the individuals affected by this system? What rights do they hold? In what ways could the AI system's outputs, failures, or biases compromise those rights? What mitigations are available? And where rights cannot be adequately protected, should the deployment proceed at all? These are governance questions of the highest order, and the AAISM professional must be prepared to contribute meaningfully to their resolution.
2.4.3 Conformity Assessments
A conformity assessment is a systematic evaluation of whether a high-risk AI system meets the requirements set out in the EU AI Act before it is placed on the market or put into service. For certain high-risk AI systems — particularly those in biometrics and critical infrastructure — conformity assessments must be conducted by independent third parties (notified bodies). For other high-risk categories, self-assessment by the provider is permitted, provided it follows the required procedures and generates the required documentation.
From a governance perspective, conformity assessments are not one-time events. Post-market monitoring requirements mean that providers of high-risk systems must continuously track performance, report serious incidents to regulators, and reassess conformity when significant changes are made to the system. This creates an ongoing compliance obligation that must be built into the AI security programme's operating model.
2.4.4 Privacy Impact Assessments
The Privacy Impact Assessment (PIA) — sometimes referred to as a Data Protection Impact Assessment (DPIA) under GDPR and similar legislation — is a structured evaluation of the privacy risks posed by a proposed data processing activity, including the use of personal data in AI systems. PIAs are mandatory under GDPR for processing activities that are likely to result in high risk to individuals' rights and freedoms, which often includes AI systems that process personal data at scale, make automated decisions with legal or significant effects, or use sensitive data categories.
A PIA for an AI system must assess: what personal data is collected and processed, and under what legal basis; how the data flows through the AI system and where it is stored; the specific privacy risks arising from the AI system's operation, including risks of re-identification, inference, or unauthorised disclosure; and the technical and organisational measures in place to mitigate those risks. Where residual risks remain high, the GDPR requires consultation with the supervisory authority before processing can commence.
2.4.5 Acceptable Risk Limits
Every risk management programme must operate within defined risk thresholds — the boundaries that separate risk the organisation is prepared to accept from risk that requires treatment. In the AI context, establishing these thresholds requires the active involvement of senior management and the governing body, because the stakes of getting them wrong — deploying an AI system that causes unacceptable harm, or failing to deploy one that would create significant value — can be very high.
Risk tolerance for AI systems should be differentiated by application domain and impact category. An organisation might accept a relatively high risk of minor inaccuracy in an AI system that generates internal meeting summaries, while tolerating near-zero risk of systematic error in an AI system that makes credit decisions or assists in medical diagnosis. These differentiated thresholds must be formally documented, approved by appropriate governance authority, and communicated clearly to the teams responsible for AI development and operation.
2.5 AI Risk Response Strategies
Once a risk has been identified, classified, and assessed against applicable thresholds, it must be addressed. AI risk response strategies follow the same fundamental taxonomy as conventional risk management — accept, avoid, mitigate, and transfer — but each strategy carries AI-specific nuances that must be understood.
| Strategy | Description and AI-Specific Considerations |
|---|---|
| Accept | The organisation acknowledges the risk and consciously chooses not to take action beyond existing controls, typically because the cost of treatment exceeds the expected impact, or because the risk falls within defined tolerance levels. Acceptance is not passivity — it requires documented justification, governance approval, and ongoing monitoring to detect if conditions change and the risk profile shifts beyond acceptable limits. |
| Avoid | The organisation eliminates the risk by declining to engage in the activity that creates it — choosing not to deploy a particular AI system, not to process certain categories of data, or not to operate in a domain where the risk cannot be adequately controlled. Risk avoidance is appropriate when the potential harm is unacceptable and no treatment option can reduce it to an acceptable level. |
| Mitigate | The organisation implements controls to reduce the likelihood of the risk materialising, the severity of its impact, or both. AI-specific mitigations include: technical controls (adversarial robustness training, input validation, output filtering, access controls); process controls (human review of high-stakes AI decisions, ongoing monitoring, regular model revalidation); and governance controls (policies, training, accountability mechanisms). Mitigation is the most frequently applicable response strategy for AI risks. |
| Transfer / Share | The organisation shifts some or all of the financial consequence of the risk to a third party — typically through insurance, contractual risk allocation with vendors, or outsourcing arrangements. Risk transfer does not eliminate the underlying risk or the organisation's operational accountability — it addresses only the financial consequences. AI-specific cyber insurance products are an emerging and rapidly evolving area. |
2.6 AI Threat Modelling
Threat modelling is a structured analytical process for identifying potential threats to a system, evaluating their likelihood and impact, and determining appropriate controls. Applied to AI, it is an indispensable tool for moving beyond generic risk awareness to a specific, actionable understanding of the attack surface of a particular AI deployment.
AI threat modelling typically follows an adapted STRIDE or similar methodology. It begins by characterising the AI system — its architecture, data flows, APIs, and integration points — and the threat actors most likely to target it (including external attackers, malicious insiders, and competitive intelligence collectors). It then systematically enumerates the threats that each threat actor might pursue against each component of the system, evaluates the likelihood and impact of each threat scenario, identifies existing controls, assesses residual risk, and recommends additional controls where residual risk exceeds acceptable thresholds.
A key insight from AI threat modelling is that the attack surface of an AI system is significantly broader than that of a conventional application. In addition to conventional attack surfaces — network interfaces, APIs, authentication mechanisms — AI systems expose training data pipelines, model weights, inference endpoints, and prompt interfaces, each of which presents unique vulnerabilities. The threat modelling exercise must encompass all of these.
- NIST AI RMF: Four functions — GOVERN (culture and policy), MAP (identify and categorise risks), MEASURE (analyse and assess), MANAGE (treat and monitor). A voluntary framework, globally applicable.
- EU AI Act Risk Tiers: Unacceptable (prohibited), High (stringent requirements), Limited (transparency obligations), Minimal (largely unregulated). Binding law for EU market participants.
- FRIA: Fundamental Rights Impact Assessment — required for deployers of certain high-risk AI systems; assesses impact on human rights.
- PIA/DPIA: Privacy Impact Assessment — mandatory under GDPR for high-risk personal data processing, including many AI applications.
- Risk Treatment: Accept (within tolerance), Avoid (eliminate the activity), Mitigate (implement controls), Transfer (insurance or contract). All four strategies apply in the AI context.
- AI Threat Modelling: A structured analytical process that maps the specific attack surface of an AI system and identifies targeted controls for each threat scenario.
AI Threat and Vulnerability Management
2.7 The AI Threat Landscape
The threat landscape for AI systems is one of the most rapidly evolving areas in contemporary cybersecurity. New attack techniques are discovered with regularity, and the defenders' task is complicated by the opacity of many AI systems — the difficulty of understanding precisely why a model behaves as it does makes it correspondingly difficult to detect and attribute attacks against it.
The AI threat landscape can be usefully divided into three categories: technical threats that target the AI system directly, non-technical threats that exploit the human and organisational context of AI deployment, and AI-enabled threats that use AI as a weapon against conventional security controls. Each category demands a different governance and control response.
2.7.1 Technical Threats Against AI Systems
Technical threats target the AI system's architecture, training process, data, or inference mechanism. The most significant categories are described below.
| Threat | Description and Mitigation |
|---|---|
| Data Poisoning | An attacker corrupts the AI system's training data, introducing malicious samples that cause the trained model to behave incorrectly. Targeted poisoning attacks can cause the model to misclassify specific inputs (a backdoor attack); untargeted poisoning degrades overall model performance. The primary goal of data poisoning is to undermine the integrity of the AI system's outputs. Mitigation: robust data validation, anomaly detection in training pipelines, data provenance tracking. |
| Adversarial Attacks (Evasion) | An attacker crafts inputs — adversarial examples — specifically designed to cause the AI model to produce incorrect outputs, while appearing normal to human observers. The classic demonstration is an image classifier that correctly identifies a panda but misclassifies the same image as a gibbon after imperceptible pixel-level perturbations are added. Mitigation: adversarial training, input preprocessing, ensemble methods, robust model architectures. |
| Model Inversion | An attacker queries the AI model repeatedly and analyses its outputs to reconstruct sensitive information about the training data. For example, an attacker might reconstruct recognisable faces from a facial recognition model, effectively reversing the privacy protections applied to the original training dataset. Mitigation: differential privacy, output regularisation, limiting query access and response detail. |
| Model Extraction (Stealing) | An attacker queries the AI model extensively and uses the responses to train a surrogate model that approximates the behaviour of the original. This enables the attacker to steal the model's intellectual property and, critically, to study the surrogate offline for vulnerabilities without triggering detection mechanisms applied to the original. Mitigation: rate limiting, watermarking model outputs, monitoring for excessive query patterns. |
| Membership Inference | An attacker determines whether a specific data record was included in the AI model's training dataset. This can enable the attacker to infer sensitive information about individuals — for example, whether a person's medical record was included in a dataset used to train a diagnostic AI. Mitigation: differential privacy, reducing model confidence output granularity. |
| Prompt Injection | An attacker embeds malicious instructions in the inputs provided to a large language model, causing it to override its intended behaviour and execute the attacker's instructions. This is analogous to SQL injection in conventional applications. In indirect prompt injection, the malicious instructions are embedded in external content that the AI system is instructed to process — a particularly dangerous attack vector for agentic AI systems. Mitigation: input validation and sanitisation, output filtering, prompt engineering guardrails, least-privilege design for agentic systems. |
| Backdoor Attacks | An attacker introduces a hidden trigger — a specific pattern in inputs — during the training process, causing the model to produce specific outputs whenever the trigger is present. The model behaves normally on standard inputs, making the backdoor extremely difficult to detect through conventional testing. Mitigation: integrity checks on training pipelines, model provenance verification, adversarial testing for trigger patterns. |
| Hallucination | The AI model generates outputs that are factually incorrect, fabricated, or internally inconsistent, with confident presentation that may mislead users into treating them as reliable. While not a deliberate attack in the traditional sense, hallucination represents a fundamental trustworthiness vulnerability that attackers may attempt to amplify. Mitigation: retrieval-augmented generation (RAG), output grounding, human oversight of high-stakes outputs. |
2.7.2 Non-Technical Threats
Non-technical threats exploit the human, organisational, and social context of AI deployment rather than the AI system's technical architecture. They are often harder to detect and harder to control than technical threats, because they exploit human psychology, institutional dynamics, and information asymmetries rather than technical vulnerabilities.
Deepfakes — AI-generated synthetic media that convincingly depicts individuals saying or doing things they never said or did — represent one of the most significant non-technical AI threats. They can be used to spread disinformation, manipulate financial markets, conduct sophisticated social engineering attacks (impersonating executives in voice calls or video conferences), and undermine the evidentiary value of audiovisual content. The governance response requires a combination of technical detection capabilities, organisational verification procedures, and public awareness.
AI-generated disinformation at scale — using language models to produce large volumes of persuasive, contextually appropriate false content — represents a related threat. Organisations may find their brand, products, or executives the subjects of AI-generated disinformation campaigns that are difficult to attribute, rapid to proliferate, and challenging to rebut. Monitoring for brand-relevant disinformation and having a rapid response capability are important governance measures.
Over-reliance on AI outputs — the tendency of users to accept AI-generated recommendations without appropriate scrutiny — is a non-technical vulnerability that manifests as a governance failure rather than a technical attack. When users consistently defer to AI outputs rather than exercising professional judgement, errors in the AI system are amplified rather than caught. Training, awareness, and process design must counteract this tendency.
2.7.3 AI System Vulnerabilities
Beyond specific attack techniques, AI systems carry structural vulnerabilities that arise from their design and operational characteristics. Understanding these structural vulnerabilities is essential for building effective AI security programmes.
Model drift — the gradual degradation of a model's performance as real-world conditions diverge from the training environment — is a fundamental structural vulnerability of all AI systems that are not continuously retrained. Left undetected, drift can cause systematic errors that accumulate over time, potentially causing significant harm before they are identified. Governance must establish drift monitoring as a continuous operational requirement, not a periodic audit activity.
Unbounded consumption — the vulnerability of AI systems, particularly large language models, to inputs that cause excessive resource consumption — can be exploited to cause denial of service. An attacker who can construct inputs that cause an LLM to generate extremely long outputs, or to enter computationally intensive reasoning loops, can effectively render the system unavailable without breaching any conventional security perimeter. Input rate limiting, output length controls, and resource consumption monitoring are key mitigations.
Excessive agency — the risk that an agentic AI system, given broad permissions to act autonomously in digital or physical environments, takes actions beyond the scope intended by its operators — is a rapidly emerging vulnerability class as AI agents become more capable and widely deployed. Governance requires that agentic AI systems operate under the principle of least privilege, with human oversight proportionate to the potential consequences of their actions.
2.7.4 AI-Enabled Threats Against Conventional Security Controls
AI is not only a target — it is increasingly a weapon. Adversaries use AI to enhance the sophistication, speed, and scale of attacks against conventional security infrastructure, creating an urgent need for AI-powered defensive capabilities to match.
AI-powered phishing and social engineering: Language models enable the generation of phishing emails and messages that are grammatically impeccable, contextually appropriate, and personalised at scale — bypassing the linguistic cues that traditionally allowed security awareness training to be effective. AI can also generate voice clones for vishing attacks and video deepfakes for executive impersonation, dramatically raising the sophistication ceiling for social engineering.
AI-assisted vulnerability discovery and exploitation: AI tools can analyse large codebases or network configurations at machine speed to identify vulnerabilities that human analysts might miss. Once identified, AI can assist in developing and testing exploits, compressing the timeline from vulnerability discovery to operational exploit. Defenders must use AI to accelerate their own vulnerability discovery and patching cycles, or risk falling permanently behind.
Automated adversarial campaigns: AI enables adversaries to conduct persistent, adaptive attack campaigns at a scale and speed that human operators could not sustain. Automated attack frameworks that learn from defensive responses and adjust tactics accordingly represent a qualitatively new category of threat that demands AI-powered defensive responses.
2.8 Mitigation Strategies for AI Threats
Effective mitigation of AI threats requires a layered strategy — no single control is sufficient to address the breadth and diversity of the AI threat landscape. The AAISM professional must understand how different control categories address different threat types and how they combine to provide defence-in-depth.
| Control | Description |
|---|---|
| Robust Data Validation | Validate all data entering AI training pipelines for integrity, provenance, and anomalies. Implement data quality gates that reject samples failing validation checks. This is the primary mitigation for data poisoning. |
| Adversarial Robustness Training | Train models on adversarially perturbed inputs so that they learn to resist evasion attacks. Regular red-teaming exercises should include adversarial attack simulations to test model robustness in practice. |
| Input Validation and Sanitisation | Apply strict input validation to all data entering AI inference systems, including prompt injection filtering for LLMs. Template-based prompting and allowlist approaches can significantly reduce the attack surface for prompt injection. |
| Differential Privacy | Apply differential privacy mechanisms during training to mathematically bound the amount of information that can be extracted about individual training records. This is the primary technical mitigation for model inversion and membership inference attacks. |
| Access Controls and Rate Limiting | Restrict access to AI model inference endpoints to authenticated, authorised users. Apply rate limiting to prevent excessive querying that might support model extraction or denial-of-service attacks. |
| Output Monitoring and Filtering | Monitor AI system outputs for anomalies — unexpected content, systematic bias, out-of-distribution responses — that may indicate compromise or drift. Apply output filters that detect and flag potentially harmful content. |
| Model Watermarking and Provenance | Embed unique identifiers in model outputs or weights that enable tracking of model ownership and detection of unauthorised use or extraction. Maintain clear records of model provenance throughout the development and deployment lifecycle. |
| Human Oversight of High-Stakes Outputs | Require human review and approval of AI outputs in high-consequence domains — credit decisions, medical diagnoses, legal proceedings, public safety. Human oversight is the most robust general mitigation for AI failures and attacks that evade technical controls. |
| Continuous Monitoring and Drift Detection | Implement real-time monitoring of AI system performance metrics, with automated alerting when metrics deviate from established baselines. Proactive drift detection prevents performance degradation from accumulating undetected. |
| AI-Specific Threat Intelligence | Subscribe to and actively use threat intelligence sources specific to AI vulnerabilities and attack techniques. The AI threat landscape evolves rapidly; governance must ensure that threat intelligence informs timely updates to controls and detection logic. |
- The concept of defence-in-depth — layering multiple controls so that no single failure compromises the entire system — is as applicable to AI security as to conventional cybersecurity. The most robust AI security programmes do not rely on any single technical control; they combine data validation, adversarial training, access controls, output monitoring, and human oversight in mutually reinforcing layers.
- An interesting tension in AI security: the same AI capabilities that make defences more powerful — anomaly detection, automated threat response, adaptive controls — also make attacks more powerful. The organisation that does not invest in AI-powered defences risks being outpaced by adversaries who are actively deploying AI in their attack toolkits.
- Consider: A sophisticated attacker embeds a backdoor trigger in a training dataset contribution to your federated learning system. Your standard validation processes do not detect it because the trigger is statistically rare and the poisoned model performs normally on standard benchmarks. How would your governance framework detect this? What changes to your training pipeline would it require?
AI Vendor and Supply Chain Management
2.9 The Enterprise's Role in the AI Supply Chain
No enterprise builds its AI capabilities entirely in isolation. The modern AI deployment draws on a complex, multi-party supply chain: foundational models developed by AI research organisations; cloud infrastructure provided by hyperscale cloud providers; data sourced from third-party data brokers and open-source repositories; software components from open-source communities; and specialised AI tools from an expanding ecosystem of AI vendors. Each link in this chain represents a potential source of risk — and a potential source of accountability — for the enterprise that ultimately deploys the AI system and bears responsibility for its outputs.
Understanding the enterprise's position in the AI supply chain is the prerequisite for effective vendor and supply chain risk management. Is the enterprise primarily an AI provider — developing and making available AI systems that others use? Or primarily an AI deployer — purchasing and deploying AI capabilities built by others? Or both? The answer determines the nature of the enterprise's responsibilities under frameworks like the EU AI Act and the practical scope of its vendor management obligations.
2.10 AI Vendor Management
AI vendor management is the discipline of selecting, contracting with, monitoring, and managing third-party AI providers in a way that protects the enterprise's risk position and ensures that the AI solutions it deploys meet applicable governance, security, and ethical standards. It is a specialised extension of conventional third-party risk management, adapted to the specific characteristics of AI products and services.
2.10.1 Key Vendor Considerations
When evaluating an AI vendor, the security professional must look beyond conventional vendor assessment criteria — financial stability, geographic footprint, support quality — to address a set of AI-specific governance questions. The answers to these questions determine whether the vendor relationship is compatible with the enterprise's AI governance framework and risk appetite.
| Consideration | Key Questions for Vendor Assessment |
|---|---|
| Security Practices | Does the vendor apply secure-by-design principles to AI development? What security testing does it conduct on its AI products? How does it respond to discovered vulnerabilities? What certifications or audit reports does it hold? |
| Data Handling | How does the vendor use customer data? Is customer data used to train or improve shared models? What data isolation controls does it apply? What happens to customer data upon contract termination? |
| Model Transparency | Does the vendor provide model cards or equivalent documentation? Can it explain how its models reach their outputs? What performance benchmarks does it publish, and how are they validated? |
| Ethical Commitments | What ethical principles govern the vendor's AI development? Has the vendor conducted bias assessments of its models? What mechanisms exist for customers to report ethical concerns? |
| Regulatory Compliance | Is the vendor compliant with applicable AI regulations, including the EU AI Act for EU-market deployments? What evidence of compliance can it provide? |
| Incident Notification | What are the vendor's obligations to notify customers of AI-related security incidents, model changes, or performance issues? What are the contractual timelines for notification? |
| Change Management | How does the vendor manage changes to its AI models — including updates, fine-tuning, or replacement of underlying foundation models? Is the enterprise notified before material model changes are made? |
| Concentration Risk | To what extent does the enterprise's AI capability depend on a single vendor? A high degree of concentration creates systemic risk — if the vendor experiences a service disruption, regulatory action, or business failure, how resilient is the enterprise's AI programme? |
2.11 The AI Deployer's Governance Responsibilities
Under the EU AI Act and other emerging AI regulatory frameworks, the distinction between AI providers (those who develop and make available AI systems) and AI deployers (those who put AI systems into use in a specific context) is legally significant. Deployers carry their own set of compliance obligations that are independent of the provider's responsibilities. This is not merely a regulatory technicality — it reflects the substantive reality that the deployer is responsible for the context in which an AI system is used, the population of individuals it affects, and the specific risks that deployment in that context creates.
For the AAISM professional, the deployer governance responsibilities most relevant to the examination include: conducting due diligence on the AI system before deployment; ensuring the AI system is used only for its intended purpose and in accordance with the provider's instructions; implementing required human oversight mechanisms; monitoring the AI system's performance post-deployment and reporting serious incidents to regulators; and maintaining required documentation throughout the deployment lifecycle.
2.12 The AI Shared Responsibility Model
The shared responsibility model — a concept borrowed from cloud computing governance — describes the allocation of security and compliance obligations between the AI provider, the cloud infrastructure provider, and the AI deploying enterprise. In AI deployments, this three-way allocation of responsibility is the norm rather than the exception, and understanding it precisely is essential for identifying and closing governance gaps.
| Party | Responsibilities in the AI Shared Model |
|---|---|
| AI Provider Responsibilities | Developing AI models that meet documented performance, safety, and ethical specifications. Maintaining model integrity and security in the provider's environment. Disclosing material information about model capabilities, limitations, and training data. Notifying deployers of material changes to models or services. Complying with applicable provider-side regulatory obligations. |
| AI Deployer Responsibilities | Selecting AI systems appropriate for the intended use case and deployment context. Configuring, integrating, and deploying the AI system securely. Implementing required human oversight mechanisms. Monitoring AI system performance in the specific deployment context. Complying with applicable deployer-side regulatory obligations, including incident reporting. Training staff on appropriate AI system use. Managing data provided to AI systems in compliance with applicable data protection laws. |
| Cloud Infrastructure Provider | Securing the underlying compute, storage, and networking infrastructure. Maintaining availability, reliability, and performance of infrastructure services. Complying with applicable certifications and regulatory requirements for cloud infrastructure (e.g., ISO 27001, SOC 2). Providing security tools and services that AI providers and deployers can use to implement their own controls. |
The critical governance risk in the shared responsibility model is the gap — the responsibilities that each party believes fall to the other, but that neither has explicitly assumed. These gaps are where AI security failures most commonly originate. The AAISM professional's role is to map the shared responsibility model precisely for each AI deployment, identify any gaps, and ensure that someone has explicitly assumed accountability for every material security and governance obligation.
2.13 Integration Risk: Legacy Systems and Intellectual Property
2.13.1 Integration with Legacy Systems
The integration of AI systems with existing legacy technology infrastructure is a significant source of risk that is often underestimated at the planning stage. Legacy systems — older enterprise applications, databases, and infrastructure — were typically designed without the API interfaces, data formats, and security architectures that AI systems require. Integrating them with modern AI capabilities creates a range of technical and governance challenges.
From a security perspective, the integration interface — the data pipeline, API connection, or middleware layer that joins the AI system to the legacy environment — represents a new attack surface that may inherit the vulnerabilities of the legacy system while also introducing new ones. A legacy system with outdated access controls, unencrypted data stores, or poor logging capabilities may compromise the security of an otherwise well-designed AI deployment. Governance must require that AI integration projects include a thorough security assessment of both the AI system and the legacy systems it interfaces with.
2.13.2 Intellectual Property Risks
AI creates intellectual property risks that flow in multiple directions. Training data used to develop AI models may contain third-party copyrighted material — text, images, code — whose inclusion was not properly licensed, exposing the enterprise to infringement liability. AI-generated outputs may themselves infringe third-party intellectual property rights if the model reproduces protected content. Conversely, the AI models that the enterprise develops or procures represent valuable intellectual property that requires protection from theft through model extraction attacks.
In the vendor context, intellectual property risks extend to the question of model ownership and portability. If a vendor hosts an AI model fine-tuned on the enterprise's proprietary data, who owns the resulting model? What happens to the enterprise's data and the fine-tuned model if the vendor relationship ends? These questions must be addressed explicitly in vendor contracts and managed as part of the AI security programme.
2.14 AI Software Supply Chain Risk
The AI software supply chain encompasses all the third-party components, libraries, frameworks, datasets, and services that flow into the development and operation of an AI system. Just as conventional software supply chain attacks — exemplified by incidents such as the SolarWinds compromise — have demonstrated the potential for devastating cascading harm from a single compromised component, AI supply chains present analogous (and in some respects more severe) risks.
2.14.1 Supply Chain Parties and Attack Surfaces
The principal parties in the AI software supply chain include: foundation model providers whose pre-trained models underlie many enterprise AI applications; open-source framework developers whose libraries (TensorFlow, PyTorch, Hugging Face) are used in AI development; training data providers whose datasets are used to build or fine-tune models; ML operations platform providers whose tools manage the AI development and deployment lifecycle; and cloud AI service providers whose infrastructure hosts AI model inference. Each of these parties represents a potential point of compromise that could propagate through the supply chain to affect the enterprise's AI systems.
Supply chain attacks against AI systems may target any of these parties. A compromised open-source library may introduce a backdoor into every model trained using it. A poisoned pre-trained model from a popular repository may carry hidden malicious behaviour into downstream fine-tuned applications. A compromised data provider may inject poisoned samples into training datasets at scale. The governance implication is that the enterprise cannot rely solely on its own security controls — it must also assess and manage the security practices of its AI supply chain partners.
2.14.2 Emerging and Evolving Best Practices
AI software supply chain security is a rapidly evolving discipline, and best practices are still being established. Several principles, however, are already clear. First, maintain rigorous provenance records for all components entering the AI supply chain — understanding where each model, dataset, and library came from and how it was created. Second, apply integrity checks to all external components — verifying cryptographic signatures, checksums, and provenance documentation before incorporating third-party components. Third, conduct security assessments of critical supply chain partners as part of the vendor management programme. Fourth, design AI systems to be resilient to component failure or compromise — using ensemble approaches, fallback mechanisms, and human oversight to reduce dependence on any single supply chain component. Fifth, monitor AI-relevant threat intelligence channels for disclosures of supply chain vulnerabilities and act on them promptly.
- Vendor and supply chain risk questions frequently present scenarios where a vendor change — a model update, a change in data handling policy, a regulatory action against the vendor — creates risk for the enterprise. The best response almost always involves contractual controls (data handling clauses, change notification requirements), ongoing monitoring (not just initial assessment), and maintaining operational resilience (avoiding dangerous dependence on a single vendor).
- Remember the deployer vs. provider distinction: the deployer cannot outsource its regulatory obligations to the provider. The enterprise that deploys a third-party AI system remains responsible for how that system affects its customers, employees, and other stakeholders.
- The shared responsibility model gap is a favourite examination topic. When a question describes an AI security failure and asks what governance control would have prevented it, look for options that address explicit responsibility allocation, not just technical controls.
2.15 Chapter Summary
Domain 2 of the AAISM examination addresses the identification, assessment, treatment, and monitoring of AI risks — together with the specific disciplines of threat and vulnerability management and vendor and supply chain risk governance. This chapter has traced the full arc of AI risk management across three interconnected sub-domains.
Part A established that AI trust is earned, multi-dimensional, and context-dependent — the foundation upon which all risk management depends. Risk identification must be continuous and structured, addressing risks across all phases of the AI lifecycle and all dimensions of potential harm. The NIST AI RMF — with its four functions of Govern, Map, Measure, and Manage — provides a practical voluntary framework for systematic AI risk management. The EU AI Act provides the binding legal architecture for the EU market, with four risk tiers whose requirements range from outright prohibition to transparency obligations. Formal impact assessments — FRIAs for fundamental rights and PIAs for privacy — are mandatory governance instruments for high-risk AI deployments. Risk treatment follows the taxonomy of accept, avoid, mitigate, and transfer, with AI-specific nuances for each strategy. Threat modelling provides the analytical depth needed to move from generic risk awareness to specific, targeted control design.
Part B mapped the AI threat landscape across three categories. Technical threats — data poisoning, adversarial attacks, model inversion, model extraction, membership inference, prompt injection, backdoor attacks, and hallucination — target the AI system's architecture, data, and inference mechanism. Non-technical threats — deepfakes, AI-generated disinformation, and over-reliance on AI outputs — exploit human and organisational vulnerabilities. AI-enabled threats use AI as a weapon against conventional security controls, raising the sophistication and scale of phishing, vulnerability discovery, and automated adversarial campaigns. The corresponding mitigation strategies — validation, adversarial training, input sanitisation, differential privacy, access controls, output monitoring, and human oversight — must be deployed in defence-in-depth layers.
Part C addressed vendor and supply chain risk — the governance challenges that arise when the enterprise's AI capabilities depend on a complex, multi-party supply chain. Vendor assessment must extend beyond conventional criteria to address AI-specific considerations: data handling, model transparency, ethical commitments, change management, and regulatory compliance. The AI deployer bears governance responsibilities that cannot be outsourced to the AI provider, including human oversight, post-deployment monitoring, and incident reporting. The shared responsibility model must be mapped precisely for each AI deployment, with explicit accountability for every governance obligation. Integration with legacy systems and intellectual property risks require specific governance attention. AI software supply chain risk demands provenance tracking, integrity verification, and ongoing supply chain partner assessment.
2.16 Practice Questions
Attempt each question before reading the answer. These questions reflect the style, depth, and reasoning demands of the AAISM examination.
Correct answer: B
The scenario describes the classic presentation of model drift — gradual performance degradation as real-world conditions (in this case, fraud patterns) evolve away from the training data distribution. The eighteen-month gap since deployment, combined with the progressive nature of the degradation, is the key diagnostic signal. Model extraction (A) would manifest as an external actor gaining model intelligence, not as internal performance degradation. Data poisoning (C) would typically affect performance from deployment, not develop progressively over eighteen months. Threshold misconfiguration (D) is a deployment issue, not a post-deployment progressive failure.
Correct answer: B
The scenario describes prompt injection — an attack in which malicious inputs cause an LLM to override its intended behaviour and execute the attacker's instructions. The defining characteristic is that user-provided inputs bypass system-level controls. The primary mitigation is input validation (preventing malicious prompts from reaching the model as intended), output filtering (detecting and blocking harmful outputs), and prompt engineering guardrails (designing prompts and system instructions to be robust against injection). Model inversion (A) concerns reconstructing training data from model outputs. Data poisoning (C) concerns corrupting the training dataset before training. Membership inference (D) concerns determining whether specific records were in the training set.
Correct answer: C
The appropriate first action is a governance and legal assessment: determining whether the clause is compatible with the enterprise's obligations under data protection law (e.g., GDPR, which requires a lawful basis for each data processing purpose, and may not permit unilateral extension to model training) and its own data governance policies. If the clause is incompatible, contract negotiation is the appropriate next step. Categorical rejection (A) without assessment is premature. Acceptance without assessment (B) ignores potential data protection violations. A penetration test (D) addresses a different dimension of vendor risk and is not the first priority when a contractual clause requires assessment.
Correct answer: B
The fundamental distinction is between a voluntary framework (NIST AI RMF) that guides organisations in how to manage AI risk through its four functions, and binding law (EU AI Act) that mandates specific outcomes and compliance requirements for AI systems in the EU market. The NIST AI RMF is globally applicable to any organisation that chooses to use it. It addresses technical and non-technical risks (eliminating A). It is not limited to federal government use (eliminating C). Neither framework is as narrowly prescriptive as D suggests.
Correct answer: B
Vendor risk is dynamic — a vendor that was assessed as secure at procurement may subsequently change its practices, experience a security incident, introduce new vulnerabilities through model updates, or change its data handling in ways that compromise the enterprise's risk position. A procurement-only assessment that is never refreshed creates a blind spot for all post-procurement changes. Contract negotiations (A) and ISO compliance (C) are valid but secondary concerns. Option D mischaracterises how regulators treat outdated assessments.
Correct answer: C
The correct first action is a risk assessment: identifying which models are affected, determining the likelihood that training environments were actually accessible to attackers, and evaluating whether any models show signs of backdoor compromise. This assessment informs the appropriate remediation strategy — which may include immediate retraining (A) or enhanced monitoring (B), depending on findings. Full disclosure (D) may be required as a downstream action if the assessment confirms compromise affecting personal data, but is premature before the scope of the incident is understood.