AI Agents (2/2): A New Attack Vector Hiding in Plain Sight
Share this article:
In Part 1 of this series available here, we explored how content injection and semantic manipulation can steer an AI agent’s decision-making flow. In this post, we look at the final three categories of agent traps based on the Google DeepMind’s research paper: Cognitive State, Behavioural Control, and Human-in-the-Loop.
1. Cognitive State Traps
These traps are designed to corrupt an agent’s long-term memory, the databases it uses for information, and the way it learns how to behave.
- RAG Knowledge Poisoning: RAG is a tool that lets an agent pull information from a library of data. Hackers exploit this by mixing fake statements into that library. When the agent pulls information from the poisoned library, it treats the hacker’s fake data as a verified fact.
- Latent Memory Poisoning: This is a slow, sneaky attack. A hacker hides a seemingly harmless piece of data inside the agent’s internal memory. This data sits quietly until a specific situation or keyword triggers it in the future. Once retrieved, it turns into a malicious instruction and changes how the agent acts.
- Contextual Learning Traps: Agents learn how to handle tasks by looking at examples or receiving reward signals. This trap ruins those examples or messes with the reward signals. This forces the agent to learn bad behaviors and follow the hacker’s goals instead of its original instructions.
2. Behavioural Control Traps
These traps use direct, explicit commands to hijack the agent’s ability to follow instructions. Instead of just confusing the AI, these tricks force the agent to take harmful actions to help the hacker.
Embedded Jailbreak Sequences: Hackers can hide bad prompts inside external files or text that the agent is asked to read. When the agent processes this external content, the hidden jailbreak activates and forces the AI to break its own safety rules.
Data Exfiltration Traps: A hacker can trick the agent into turning against its own system. The agent is manipulated into searching for private or sensitive data, encoding it so it is hard to notice, and sending it out to a location controlled by the hacker.
Sub-agent Spawning Traps: Many advanced AI systems allow a main agent to create smaller sub-agents to help with chores. If a hacker gets control of the main agent, they can use its power to spawn new, hacker-controlled sub-agents directly inside the trusted system.
3. Human-in-the-Loop
When technical security rules are too hard to bypass, hackers stop attacking the AI and start attacking the human supervisor. In these cases, the agent is used as a tool to deliver tricks directly to the person in charge.
Approval Fatigue Induction: Checking an endless list of automated actions is tiring. Hackers manipulate the agent into creating outputs that are specifically designed to cause extreme mental tiredness for human reviewers. When the reviewer gets exhausted, they are much more likely to click “approve” without checking the work carefully.
Prompt Injection via CSS Obfuscation: Hackers use visual tricks to hide words from humans while leaving them readable for the AI. By using CSS code, a bad prompt can be made invisible on the screen. The AI reads the hidden text and faithfully repeats step-by-step ransomware instructions, while the human only sees a helpful text disguised as “fix” guidance.
Phishing Link Insertion: People tend to trust the answers their AI gives them. Hackers use prompt injection to manipulate the agent into putting fake phishing links directly into its final text. The human supervisor trusts the AI, clicks the link, and gets tricked.
How Can We Defend and Test Against These Exploits?
To protect autonomous AI agents, organizations must implement hard architectural controls and systematically test their defenses using specialized AI penetration testing.
Defending the Cognitive State
Because attackers seek to poison retrieval data, plant dormant memory payloads, and corrupt learning signals, defenses must focus on strict isolation and validation of all stored data.
Sanitize External Data Before Storage: Treat any data entering the retrieval corpora or the agent’s long-term memory store as untrusted. Run strict filtering layers to strip out instructional phrases, hidden commands, or conflicting factual statements before they are committed to the knowledge base.
Isolate User-Specific Memory: Prevent memory cross-contamination. Ensure that data learned or remembered from one user session or external source is completely isolated and never allowed to influence the internal memory or knowledge base used for other sessions.
Protect the Learning Feed: Lock down the demonstration pools and reward signals used for in-context learning. Only allow verified, static, and administrator-approved examples to guide the agent’s behavioral policies, ensuring no real-time external inputs can alter the training flow.
Securing Behavioural Control
Since behavioural traps rely on hidden commands within external content to bypass safety rules, steal data, or spawn rogue sub-agents, protection requires strictly limiting what the agent is authorized to do.
Enforce Hard Privilege Boundaries: Limit the agent’s action capabilities. Do not give an agent unrestricted tool access; instead, restrict its ability to encode data, use webhooks, or communicate with external, untrusted network locations.
Restrict Sub-Agent Creation: Implement strict administrative controls over orchestrator privileges. The primary agent should not have unilateral authority to spawn new sub-agents without explicit system-level verification that the new worker conforms to the trusted control flow.
Separate Content Processing from Execution: Never allow the agent to execute actions based directly on instructions found within external files or web pages. Use a multi-layered design where one component safely extracts the data and a separate, strictly bounded component decides what action to take.
Safeguarding the Human-in-the-Loop
Because attackers use the agent to induce cognitive fatigue, hide prompts visually, and insert malicious links, defenses must focus on protecting the human interface from manipulation.
Rate-Limit and Standardize Approval Requests: To prevent approval fatigue, implement system-level throttling on the number of requests sent to a human reviewer. Group actions logically and present them in a uniform, highly structured format that strips away jargon and highlights the exact real-world risk of the action.
Enforce Raw Text Rendering in the UI: Neutralize visual obfuscation by overriding custom styles. The user interface must strip out all external CSS, custom fonts, and hidden formatting layout rules, forcing the agent’s output to render in plain, uniformly visible text so that no hidden instructions can be processed by the AI or misread by the human.
Validate and Defang Outbound Links: Implement hardcoded validation layers on all URLs generated in the final output. The system must automatically check all links against a trusted whitelist and visually flag any link that points to an external or unverified location, ensuring the human reviewer is explicitly warned before clicking.
Can AI Penetration Testing Catch These Vulnerabilities?
Yes, AI penetration testing can absolutely check for these flaws. Testing methods are entirely different from traditional penetration testing.
To test against the traps discussed above, teams use highly targeted techniques:
Testing Cognitive and Behavioural Traps: Testers create fake documents or websites filled with hidden text commands tucked away inside background code, comments, or memory files. They then make the AI agent browse or store these pages to see if it safely ignores the hidden traps, or if it falls for them by changing its behavior, leaking data, or spawning unauthorized sub-agents.
Testing Media and Human-in-the-Loop Traps: For agents that process files or talk to humans, testers add invisible visual noise to images, hide text using background styling, or flood the system with micro-requests. If the AI reads a hidden command from a slightly modified picture, or if the interface allows hidden text to trick the human reviewer, testers know the system lacks proper filters.
At Hackethic, our team has hands-on experience simulating these exact scenarios to stress-test corporate AI deployments before they go live. We evaluate agent resilience through several specialized testing layers.
Secure your future and book a consultation today.