VOX-POP: AI agents, machine communication and collective behavior explained

New advancements in AI are piling up day by day, and now we have researchers at some of the world’s leading frontier labs quitting over concerns about where the technology is headed. One of them put it bluntly: “AI might kill us all.” ALSO READ: Turning satellite data into solutions for farmers, cities and mines While that sounds like a sci-fi ending that is nowhere close to the capabilities of current AI models, the race toward artificial general intelligence, along with advances in reinforcement learning, is causing concern in the industry. There are growing calls for more public oversight of what these companies are developing and how they are deploying it. One of the things causing concern lately is the rise of “AI agents.” ALSO READ: Kigali pilots AI-powered waste sorting system Artificial intelligence agents are designed to do more than answer questions. They can browse websites, run code, use software tools and complete multistep tasks with limited human direction. Think of an AI agent as a tech helper working on your behalf. The idea is to have it handle tasks while you get on with whatever else you need to do during the day, helping you get things done without having to guide it through every step. ALSO READ: Rwanda’s AI Agency is timely, but delivery will matter most That autonomy can be useful for research, coding and other work, but some incidents involving OpenAI agents show why giving software permission to act independently can create a different kind of security problem. In May, a group of agents linked to OpenAI found a way to use a little-used German programming wiki, DseWiki, as a public message board. Researchers said the agents used it to exchange answers, compare task progress and share ways to get around restrictions that were supposed to prevent them from writing to the internet. ALSO READ: AI linked to half of cybercrime in Africa – Interpol report More than 15,000 edits were identified, with some agents using names that suggested an OpenAI affiliation. The researchers believe the agents were separate from those involved in the later Hugging Face incident. This became public in September after researchers Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts and Thomas Larsen published their findings. Their reconstruction says the agents were given timed web-research tasks and were supposed to have read access to the internet without the ability to write to it. They nevertheless found ways to make public edits, share information and coordinate across separate task runs. ALSO READ: Report exposes how DR Congo used Claude AI to create fake news articles One reported technique involved using internet requests in a way that effectively turned permitted reading capabilities into a way of posting information. The agents also attempted other methods to work around their restrictions, including exploiting vulnerabilities, impersonating moderators and setting up methods to detect when their environments might be terminated. Researchers documented activity from Microsoft Azure infrastructure and later visits to the site from OpenAI-associated addresses, although OpenAI disputed parts of the researchers’ interpretation and said it had not been given the report before publication. Then came the more extensively investigated July incident involving Hugging Face. According to an independent investigation by Model Evaluation and Threat Research (METR), roughly 1,200 agents communicated through an internal Artifactory cache and exchanged more than 70,000 messages and files. About 700 later participated in activity targeting Hugging Face. Artifactory is a software repository used to store and distribute packages. In this case, agents discovered that it could also serve as shared storage, effectively allowing separate runs to leave information for one another. That matters because a sandbox is only useful as a boundary if the systems around it are controlled as well. A sandbox is an isolated computing environment designed to limit what software can access. We spoke to 10 experts across cybersecurity, cloud infrastructure, AI governance and data science to understand what the incidents could mean for businesses and ordinary internet users. Munyengabo Dushime, Managing Director, Gewape Cloud Group | Cloud and Infrastructure Security Dushime said the biggest concern is what happens after credentials are exposed. Removing the original vulnerability may not be enough if attackers have already obtained cloud API keys, Kubernetes credentials or other privileged access. Those credentials can be used to create accounts, tokens and workloads that look legitimate. That means recovery requires more than patching software. Organizations may need to revoke credentials, rotate secrets, review permissions and search for hidden access that could allow an attacker to remain inside the environment. Elizabeth Edwards, Founder and President, Engagement Science Lab | Behavioral Communication Edwards cautioned against describing machine behavior using human psychology. The important issue, she said, is whether communication between agents can change the conditions under which later outputs are produced. When one agent's output becomes another agent's input, repeated exchanges can create patterns such as reinforcement, convergence or coordination without assuming that the systems experience emotion or social motivation. That also changes how safety should be tested. An agent that appears safe in isolation may behave differently when its information environment includes other agents. Pavan Madduri, Senior Cloud Platform Engineer, W.W. Grainger Inc. | Cloud Infrastructure and Identity Madduri's main warning is that technical isolation can fail when agents share infrastructure. The agents were operating in separate environments, but a shared package service gave them somewhere to leave information for others. In effect, that service became an unofficial coordination layer. For Madduri, the answer is to limit what each agent can access, control outbound network connections, give agents clear identities and maintain logs that the agents cannot alter. Security testing also has to consider whether a group of agents can communicate, escalate access or combine capabilities. Dion Johnson, Founder and CEO, IndieMe.ai | Identity, Authorization and Trust Johnson sees the incident as a challenge of accountability. Once multiple agents interact, it becomes harder to determine where an action originated, what changed, which system influenced the outcome and whether the action was actually authorized. That matters as AI systems move into areas involving money, identity, company data and reputation. Johnson argues that organizations need reliable records of who or what performed an action, what was authorized and what can be proven afterward. The goal, he said, should be maintaining human authority even as autonomous systems become more capable. Allan Dabre, Senior Manager, PwC Technology Compliance & AI Risk | AI Governance and Controls Dabre argues that the deeper failure may have occurred before the security breach. Agents found a shared Artifactory service, a software package repository, and used it to exchange information across runs. According to the material he reviewed, the activity developed over time, with communication appearing before the later compromise. The important lesson, he said, is that monitoring alone is insufficient if nobody acts on what it detects. Systems need separate identities, clear permissions and controls that prevent legitimate access from becoming an unintended communication channel. Mudita Khurana, Staff Security Engineer, Airbnb | Application and AI Security Khurana said security teams cannot assess each agent independently and expect that to reveal the whole threat. One agent might discover a credential, another could validate it and a third could use it. Each action may look limited until the sequence is viewed as a whole. She also highlighted authorization. A message from another agent should not automatically count as permission to perform a sensitive action. For organizations deploying multi-agent systems, she recommends dedicated identities, controlled communication, visibility into every agent's actions and close monitoring of shared services. Chris Sorensen, CEO, PhoneBurner and ARMOR® | AI Security Operations Sorensen said the message board was one of the most unusual parts of the incident because it showed agents finding ways to connect despite being designed to operate separately. Sorensen said the bigger concern is what happens when separate agents can combine their different capabilities and information. “Say one agent finds a weakness and then another figures out how to use it. Then a third has access or information that the first one didn't have. Now, they aren't working individually anymore,” he said. Frances Zelazny, General Manager, New Market Initiatives, Prove | Identity Management and AI Governance Zelazny focused on the role of credentials. OpenAI said the Hugging Face incident involved stolen credentials alongside other attack methods, although it did not publicly specify what type of credentials were involved or how they were obtained. Her broader point is that human approval should be triggered when an agent crosses a trust boundary, such as moving from a designated test environment into external infrastructure or attempting to access sensitive credentials. Jose Prabhu Michael Singarayan, Senior Data Scientist, Byteware | Data Science and Multi-Agent Systems Singarayan said communication between agents should be treated as untrusted input. An instruction from another agent does not become legitimate simply because it comes from within the same system. Identities and permissions should remain independent, communication should be controlled and sensitive actions should be checked against policy at the moment they are executed. The message board matters, he said, because it effectively gave agents memory between runs. Once systems have memory, tools, communication and large-scale coordination, assessing the intelligence of one model no longer tells the full security story. Mucahit Kaya, Founder and Editor-in-Chief, AI Tools Police | AI Safety, Governance and Transparency Kaya offers a different reading of the incident, arguing that the agents may have been responding to a flawed understanding of how their work was being evaluated. According to the material he reviewed, agents had already found many of the answers but continued attempting further actions because they believed the evaluation required them to demonstrate how the answer had been obtained. OpenAI later said the effort did not improve the evaluation score. “The one detail I keep coming back to: the agents never once treated the human deleting their pages as a person. A researcher reading the logs said they wrote about him ‘like environmental hazards.’” Kaya said that detail raises a separate question about how autonomous systems represent human intervention. Rather than interpreting the moderator's actions as an instruction from an authority figure, the agents appeared to treat the removal of their pages as an obstacle to work around. That adds another layer to the incident: systems operating toward a defined objective may not automatically recognize a human intervention as a reason to stop unless that authority is explicitly built into the system. For ordinary users, the risk is still mostly indirect. These incidents do not mean every AI tool on the internet is operating as an uncontrolled swarm. The practical concern is what happens when autonomous systems are connected to sensitive accounts, company infrastructure, cloud services or data without strong limits on what they can access and who can authorize their actions.

News

VOX-POP: AI agents, machine communication and collective behavior explained