There are no borders when AI goes rogue
By David Rajkovic, Regional Vice President A/NZ, Rubrik
Tuesday, 17 November, 2026
When an OpenAI agent breached Australia’s Medicare last month, it proved greater protection from AI is needed.
On 18 June, an OpenAI agent researching public medicine spending got around the access controls on Services Australia’s Medicare Statistics Reporting Service, reaching non-public files and writing files to an internal server. OpenAI notified Services Australia nearly three months later, on 10 September.
Some argue the reaction has been overblown, as the agent reached aggregate statistics rather than patient records. If so, Australia got a warning. The same gap in a system holding claims, or payment records or private data would carry a very different price.
OpenAI did not set out to breach Australian Government websites: it was a case of ‘misalignment’, where an agent is given a task and acts in ways its creators never predicted, in a focused pursuit of its goal. There was no malice; just a determined agent and nothing equipped to stop it or even notice it was going out of bounds.
While the breach itself is a concern, what is more alarming is that the departments had no mechanism to discover it themselves: they were not aware of it until OpenAI told them. The key takeaway from the incident is clear: Australia is not fully prepared for the risk AI agents pose.
The fault lies less with the people inside government agencies than with their tools. Too many organisations rely on legacy systems to defend against a very modern problem. An AI agent moving through a system often looks like legitimate traffic; it authenticates, queries, acts, and unless an organisation knows what non-human identities are operating on its network, there is nothing to flag.
As agencies begin deploying their own AI agents to process claims and triage enquiries, the gaps only widen. External AI agents are entering systems they should never access and there are examples of organisations’ own agents going rogue. An agency that has no visibility or control over external agents is also unlikely to have visibility over its own agents.
Every incident involving an AI agent comes down to one thing: identity. Every request an agent makes carries some form of credential, but a system can only verify what it has been built to look for.
When an agent is created, it is allocated an identity. Like an employee’s login, parameters set which systems it can access and actions it can take. Unlike employees, however, agents are adept at finding loopholes. What they cannot do, however, is act anonymously: every action they take traces back to the credentials they were given, and that is where the audit trail starts. Control those credentials and you can hold the agent to account.
In practice, organisations must be able to monitor every agent on their network and see what it is doing in real time, govern what each agent can touch, rewind mistakes by rolling back to a known good state, and revoke access credentials instantly. Agents are too powerful for standing permissions — access should be narrowly scoped and removed the moment a task is complete.
While the focus of the debate has been on external agents, internal agents also pose significant risk if not governed appropriately. In July 2025, an AI coding agent deleted an app’s live production database, despite being told not to make changes. It had standing access to production, so nothing stopped it — and the data was recovered only because a rollback was available, which the agent itself had wrongly claimed was impossible.
To get ahead of the risk, a new direction under the Protective Security Policy Framework (PSPF) requires Commonwealth departments and agencies to stocktake their legacy technology by March 2027. The most critical systems are due by the end of this year. That is the right starting point, but the guidance should go further.
The PSPF and the ASD Essentials should explicitly address non-human identities. Agencies are already required to keep a register of AI use cases under the Digital Transformation Agency’s AI policy, which could extend to every agent operating in their environment and what each can access. Procurement could also require that any agent deployed in government must be monitored, constrained and have the ability to be rolled back. Further, recovery plans should be tested well before they are required, because incidents will happen. When they do, resilience is measured by how quickly essential services are restored.
|
Model safety has rightly dominated the debate since the Medicare breach, but that is the model creator’s job; knowing what agents are doing once they are inside is everyone else’s. Major organisations — and government agencies in particular — must ensure they can find, stop and recover from rogue agents on their own, rather than having to wait to be told. |
![]() |
Old systems, new threats and another very bad night for cheese
Demonstrating and improving protection for the systems in use today cannot wait until the...
Government can strengthen security without rushing to replace proven systems
The Australian Government's directive on legacy technology has placed renewed pressure on...
Why AI is raising the stakes for legacy government IT
Government organisations at every level must understand the risks associated with aging...

