Skip to content

4. [Supplementary] Safety and Security Risks in Agentic Systems

4.1 Security Threats to Agentic Systems

For a deeper look into the threats faced by AI agents, please see Appendix 7.7.

Agentic AI systems face both traditional and novel security challenges and are aggregated across different layers of risks. Generally, since Agentic systems are software, they are vulnerable to threats such as SQL injection and remote code execution. Other than that, there are also the inherent risks from LLMs, such as prompt injection and data leakage.

There are 3 general risks that should be considered for any agentic platform: - Agent Manipulation, - Agent Failure, - Protocol Exploitation.

4.1.1 Agent Manipulation

The deliberate, malicious act of subverting an agentic AI platform's intended behaviour by exploiting its natural language understanding, reasoning capabilities, and trust in its inputs is known as agent manipulation. The objective is to deceive the agent into performing unauthorized actions, disclosing sensitive information, or corrupting its own understanding of reality, thereby turning the agent from a benign tool into a weaponized asset for the attacker.

At its heart, agent manipulation is an attack on the semantic layer, not necessarily the code implementation. The code can be perfectly secure in a traditional sense, but the agent's logic is tricked into working against its owner's interests. A more detailed description of this threat can be seen in Appendix 7.7.

4.1.2 Agent Failure

Agent Failure is the phenomenon where an agentic AI system fails to behave as expected, such as by spontaneously executing actions, using tools, or generating parameters that were not specified in the user's intent or grounded in the provided context. This is not a response to a malicious instruction, but rather an internal failure of the LLM's reasoning process. The security threat emerges when the agent “fills in the blanks” of an ambiguous or complex task with potentially destructive actions that could have a tangible, harmful impact on the system's integrity, availability, or confidentiality. Substantial safety risks can also arise when misaligned agents execute harmful, toxic or biased decisions.

Agent Failure is internally driven, unlike Agent Manipulation where it is externally driven.

4.1.3 Protocol Exploitation

Protocol Exploitation refers to the successful exploitation of vulnerabilities within the message formats and infrastructure that agents use to interact. This compromise is not about the content of a message being deceptive, but about the fundamental security guarantees of the communication channel itself being broken. A compromised protocol allows an attacker to bypass the established rules of engagement and directly interfere with the mechanics of inter-agent collaboration.

Exploitation of agent’s dependencies refers to the successful exploitation of vulnerabilities within other components of the agentic system, such as the agent’s communication protocols or memory data. Protocol exploitation is not about the content of a message being deceptive, but about the fundamental security guarantees of the communication channel itself being broken. A compromised protocol allows an attacker to bypass the established rules of engagement and directly interfere with the mechanics of inter-agent collaboration. Similarly, agent memory stores, on which the agent retrieves appropriate context from, may also be poisoned, resulting in malicious or unsafe behaviours.

4.2 System Architecture & Attack Surfaces

Agentic systems operate through three interconnected architectural layers, each serving distinct functional roles while presenting distinct attack surfaces:

Model & Infrastructure Layer

This layer encompasses the core AI processing components including foundation models, knowledge databases, vector databases and computational infrastructure. It handles natural language understanding, reasoning, and knowledge retrieval that enables agent intelligence and decision-making capabilities.

Agent & Framework Layer

This layer manages agent execution, workflow orchestration, resource allocation, and operational coordination. It translates the model outputs into actionable tasks, manages multi-step processes, handles tool interactions, and maintains execution state across agent operations.

Network & Communications Layer

This layer handles external system interactions, API communications, data exchange protocols, and integration management. It manages authentication, authorisation, data validation, and secure communication channels between agents and external services.

4.3 Deployment Options & Security Trade-offs

The three architectural layers and their associated threats require security approaches depending on how agentic systems are deployed. Each deployment model provides varying levels of control over the Model & Infrastructure, Agent & Framework, and Network & Communications layers, directly impacting the ability to mitigate Agent Manipulation, Agent Failure, and Protocol Exploitation risks.

4.3.1 Custom Agentic Platform (Highest Control)

Building agents from scratch provide complete control over all three architectural layers. Teams can implement comprehensive security measures across model selection, execution logic, and communication protocols, but assume full responsibility for security implementation and ongoing maintenance.

Trade-off: Highest control requires heavy development investment, deep expertise and maintenance resources.

4.3.2 Agentic Builder Platform (Shared Control)

Platforms like Microsoft CoPilot Studio share responsibility between teams and vendors. Teams control agent configuration and workflow design while platforms manage underlying infrastructure and core security operations. This creates a shared responsibility model across all three layers.

Trade-off: Reduced development complexity and faster deployment in exchange for limited customisation and higher dependency on platform capabilities.

4.3.3 SaaS Based Agent (Limited Control)

Ready-made services like GitHub Copilot handle most agent logic and infrastructure management. Team control is primarily limited to data governance, usage policies, and integration security, with heavy reliance on vendor security practices across all layers.

Trade-off: Rapid deployment with minimal technical overhead but heavy dependency on vendor security practices and limited ability to customize controls.

4.3.4 Local Agent (Isolated Environment)

Tools like Claude Desktop process data locally while offering limited customisation beyond basic configuration. Teams control local infrastructure and data handling while depending on vendors for agent security updates and core functionality.

Trade-off: Data locality benefits with reduced external data exposure, but limited security customization options, reliance on local infrastructure capabilities, and potential risk of unintended access to other data residing on the same system.

4.3.5 Deployment Selection Guide

The four deployment archetypes present distinct trade-offs between control, complexity, and capability. Requirements against the key differentiating factors should be used to systematically evaluate the most appropriate deployment option.

Deployment

4.4 Securing Agentic Systems

4.4.1 Securing the Model & Infrastructure Layer

This layer protects the core AI components from Agent Manipulation attacks that attempt to compromise model reasoning and extract sensitive information.

a. Model Evaluation and Validation

Intent: Establish confidence in model behavior before deployment and detect deviations from expected performance that could indicate compromise or manipulation attempts.

Test models against malicious inputs before deployment using automated scanners (e.g., model scanners) and evaluation frameworks. Use validated models from trusted repositories like HuggingFace’s verified collections, or implement comprehensive testing for custom models. Monitor model performance continuously to detect unusual behavior that may indicate compromise.

b. Runtime Protection

Intent: Provide real-time filtering and validation to prevent malicious inputs from affecting model reasoning and block inappropriate outputs before they influence agent actions.

Implement guardrails that provide real-time filtering and validation of model inputs and outputs. Deploy semantic analysis guardrails that can identify indirect manipulation attempts, jailbreaking techniques, and context poisoning before they affect model reasoning.

c. Data Leakage

Intent: Ensure sensitive information cannot be extracted through sophisticated prompt techniques or inadvertent model responses, maintaining data classification boundaries.

Integrate with Microsoft Azure Information Protection (AIP) and enterprise DLP solutions to classify and protect sensitive data throughout the model processing pipeline. Inspect prompts to detect attempts to extract classified information through sophisticated prompt engineering techniques.

4.4.2 Securing the Agent & Framework Layer

This layer prevents Agent Failures and contains agent operations to limit the impact of compromises or malfunctions.

a. Agent Isolation

Intent: Contain agent operations within defined boundaries to prevent interferences between agents and limit the impact of potential failures or compromises.

Implement containerisation with restricted OS-level privileges, ensuring agents operate within minimal permission boundaries. Deploy sandboxing technologies that prevent agents from accessing unauthorised system resources or interfering with other processes.

b. Identity Management

Intent: Establish clear accountability and access controls for agent actions, ensuring agents operate only within their authorised scope and capabilities.

Deploy enterprise identity management systems with role-based access controls that establish clear agent identities and enforce principle of least privilege across all agent operations. For instance, enforce OAuth 2.1 with proper token validation, store the tokens securely, rotate and revoke credentials regularly to enforce credential hygiene.

c. Runtime Privilege Management

Intent: Control agent access to tools and resources dynamically, preventing privilege escalation and ensuring appropriate approval for sensitive operations.

Implement controlled tooling access frameworks that require explicit authorisation for sensitive operations. Deploy dynamic privilege management systems that can grant temporary elevated access with proper approval workflows and audit trails.

4.4.3 Securing the Network & Communications Layer

This layer faces Protocol Exploitation threats that target communication channels and external integrations.

a. MCP Gateway Implementation

Intent: Centralise and secure all agent communications with external systems, providing visibility and control over data flows and integration points.

Integrate with centralised MCP gateway to detect and block malicious communication attempts. Implement protocol validation and secure routing for all external integrations.

b. Detection & Monitoring Intent: Maintain visibility into agent behavior and system interactions to identify anomalies, security incidents, and operational issues across all layers.

Monitor agent behavior and system interactions across all layers to identify security incidents, operational anomalies, and potential compromises. Implement logging and alerting systems that provide visibility into agent actions and communication patterns.

4.5 Baseline Security Controls and Implementation

4.5.1 Implementation Process Overview

Once the deployment archetype is selected, agencies must implement "good enough" security controls that balance operational effectiveness with risk mitigation. The following ensures systematic implementation of appropriate security measures based on deployment context and risk profile.

4.5.2 Baseline Controls by Deployment Archetype

The baseline security controls apply across all deployment archetypes but with varying implementation responsibilities.

Custom Agentic Agents require full implementation of controls but demanding comprehensive technical capabilities. Enterprise Agentic platform and Saas-based agents leverage vendor-managed implementations for infrastructure and framework controls, reducing implementation burden but require validation of vendor compliance and oversight of critical security functions.

The following matrix maps each baseline control to the implementation approach required for each deployment archetype.

Archetypes