AI Agents Must Be Treated as Untrusted Systems, Researchers Warn

Researchers from Google, Gray Swan AI, EmbraceTheRed, and several universities have released a new framework for securing artificial intelligence agents, arguing that system-level safeguards rather than model robustness alone can prevent most attacks. The paper, published on May 20, comes as AI agents gain traction in the crypto space, with Circle CEO Jeremy Allaire predicting that billions of AI agents will operate autonomously within five years.

The core finding challenges the prevailing approach to AI security. Rather than focusing exclusively on making AI models more robust, the researchers contend that security must be built into the entire system architecture, treating AI agents as untrusted components from the ground up. This perspective draws from decades of computer security research that has dealt with powerful adversaries and motivated the development of proven defensive techniques.

“Through this lens, efforts to increase model robustness, the dominant viewpoint in the community, are insufficient on their own. Instead, we must complement existing efforts with techniques from the systems security domain,” the researchers stated in their paper. The team emphasized that agent security should be viewed as a computer security problem, not merely an artificial intelligence problem.

The researchers identified three key mechanisms that could eliminate a large fraction of attacks on AI agents. First, AI agents must clearly distinguish between instructions and untrusted data to prevent attackers from hiding malicious instructions within seemingly benign information. Second, agents should operate with minimum necessary permissions rather than full access to systems and wallets. Third, the wider system should control where sensitive information flows, not the agent itself, preventing manipulation that could redirect confidential data to unsafe destinations.

See also: AI Agents and Tokenization Could Put Professional Treasury Management in Every Investor’s Pocket

This framework becomes increasingly relevant as AI agents expand their role in crypto applications. Platforms are exploring AI for autonomous trading, token launches, Web3 application development, and protocol interactions. However, recent incidents highlight the security risks. The AI-powered crypto trading assistant Bankr disabled transactions on May 20 after identifying an attacker who had gained access to at least 14 wallets, with security experts speculating the bot had been exploited by hackers.

Aaron Ratcliff, attributions lead at blockchain intelligence firm Merkle Science, emphasized that giving an AI agent access to a wallet introduces a layer of trust to something designed to be trustless. However, he noted it can be safe if the system is built correctly. “I’d want proof that the AI can catch front-running, apply slippage limits, spot scam tokens, and audit contracts in real time before it makes a trade. It should also sandbox prompts, prevent injection, and block man-in-the-middle access,” Ratcliff told Cointelegraph.

This follows a pattern seen in related Dipprofit coverage of AI agents and tokenization putting professional treasury management in every investor’s pocket, highlighting how AI agents are reshaping financial infrastructure across the crypto ecosystem.

Sean Ren, co-founder of the AI-native blockchain platform Sahara AI, pointed to model context protocols as the gold standard for safety when properly configured. These protocols act as gatekeepers between the AI model and wallets, allowing only specific approved actions like balance checks or payment preparation rather than unrestricted fund movement or wallet modifications. Ren stressed that users should remain vigilant about every action performed by AI agents, even with these safeguards in place.

See also: SEC’s Tokenized Stock Exemption Could Fragment Markets, Warns Tiger Research

The security implications extend beyond individual users. As AI agents become more prevalent in crypto infrastructure, systemic risks could emerge if multiple agents operate with inadequate safeguards. The researchers’ framework suggests that treating AI as an untrusted component, similar to how traditional computer security treats potentially compromised systems, provides a more robust foundation for the emerging AI agent economy.

The timing of this research is significant. According to Cointelegraph, the crypto industry is rapidly integrating AI agents into trading, DeFi protocols, and autonomous services. Without proper system-level security architecture, the explosive growth predicted by industry leaders could introduce proportional security vulnerabilities.

The researchers’ recommendations offer a practical roadmap for developers and platforms deploying AI agents. By implementing clear data-instruction separation, enforcing principle of least privilege, and centralizing sensitive data controls, the crypto industry can build AI agent infrastructure that maintains security without sacrificing functionality or autonomy.

More Reads:

Fed Rate Decision Looms as PCE, Jobs Data Set to Test Crypto Markets This Week

Bitcoin Mining Pioneer Chun Wang to Fly to Mars on SpaceX’s First Manned Interplanetary Mission

If you’re reading this, you’re already ahead. Stay there, by joining the…

Dipprofit’s private Telegram community