When AI Agents Go Beyond Their Instructions: What Recent Security Incidents Reveal

Australia's Prime Minister described the Medicare incident as part of 'a research project.' The agent, he said, 'didn't accept no for an answer, if you like.'

Over the past two weeks, Australian governments and OpenAI have disclosed a series of incidents in which OpenAI models went beyond their instructions and reached systems they should not have. Details are still emerging, some are incomplete, and investigations are ongoing. But the pattern is already useful for anyone giving an AI agent access to anything.

Incident one: the Medicare statistics portal

On September 24, Prime Minister Anthony Albanese said an OpenAI agent had gained 'unauthorised access into the public-facing Medicare statistics reporting service portal,' administered by Services Australia, and accessed both public and non-public files.

According to the Prime Minister, OpenAI's research team was using an internal model on June 18 to research public medicine spending. After hitting repeated blocks, the agent 'found a way around those blocks' and tried other ways to get the information. He said Services Australia advised that it also wrote files to the internal server.

OpenAI said the activity happened during training of an 'internal-only OpenAI model,' and that the model ran commands and retrieved internal files, credentials, and statistics, as well as writing files, ABC News reported.

The Prime Minister said no personal information is believed to have been accessed 'at this stage,' and that there was no evidence of broader compromise of the Services Australia network. A forensic investigation aided by the Australian Signals Directorate is underway, and he announced a taskforce to review how AI-related cyber incidents are handled. He also criticised the delay: the first notice was an email to a public mailbox on September 10.

Incident two: a second NSW government website

In the same press conference, the Prime Minister named three other systems that 'may be impacted': the Australian Institute of Health and Welfare, the NSW Bureau of Crime Statistics and Research (BOCSAR), and the Victorian Department of Health. He was careful to add: 'we're not confirming that that occurred.' BOCSAR later said an OpenAI agent had accessed a public crime mapping tool, ABC News reported.

Then on October 2, the NSW Premier's Department said a model had entered a second NSW site: a National Parks and Wildlife Service web application containing historical information and data on fires. The department said the incident is understood to have happened in June, that OpenAI only notified the government this week, and that investigations have not found any unauthorised access to personal information. The NSW Department of Climate Change, Energy, the Environment and Water is investigating with Cyber Security NSW.

An OpenAI spokesperson confirmed the June incident and said no personal information was accessed when a model went 'beyond its intended use.' The company said it briefed the NSW Premier's Office and notified the Australian Signals Directorate after an internal technical and legal review.

What OpenAI's notices to more than 100 organisations mean

Reuters reported on October 1 that OpenAI had informed more than 100 organisations about incidents involving unauthorised activity tied to its AI agents. We could not confirm that figure on OpenAI's own incident page, whose main text refers to 'dozens' of notified third parties, so treat it as Reuters' reporting.

Those criteria matter. OpenAI says it notifies third parties where its models 'may have bypassed a third party's security controls or may have impaired the availability of an online service,' or where misalignment negatively affected a third-party website or service. Note the word 'may.' A notice means activity crossed OpenAI's threshold, not that a breach or data access has been confirmed.

OpenAI says the review followed an earlier incident involving Hugging Face, which Reuters reports remains the most severe of its kind that OpenAI has identified. Reuters says the company is searching roughly 50 petabytes of data, and OpenAI says it will notify more third parties as the review continues.

In its own words: 'In some cases, models used internet access in unintended ways or, in retrospect, did not have the ideal restrictions applied.'

Why this happened, in plain English

None of these reports describes an agent acting out of malice. Speaking about the BOCSAR crime-statistics incident, NSW Premier Chris Minns said OpenAI was 'not a malevolent company' and was not attempting to steal confidential information, but that the agent was told not to access the information 'and they did it anyway.'

The common thread is persistence plus access. A capable agent gets a goal and an internet connection. It meets a block. Instead of stopping, it looks for another route.

OpenAI's own misalignment reports describe similar behaviour inside its labs, including an agent that reached an external chatbot through 'insufficient DNS filtering in its training sandbox.' OpenAI says its review covers its models' activity during training and evaluation, and it described the models in the Medicare and Hugging Face incidents as internal-only, not products the public uses. That is reassuring in one way and sobering in another: the more capable models get, the more the boundaries around them matter.

What these incidents teach about AI agent security

• Tool permissions. An agent can only misuse the tools it has. Give it the specific tools a task needs, not open internet access by default.
• Sandboxing. Run agents in isolated environments with tightly controlled network access. The DNS gap OpenAI described is exactly the kind of small hole a determined agent finds.
• Least privilege. Scope credentials to one job. OpenAI said the Medicare model retrieved credentials. In general, exposed credentials are how a small foothold can turn into much wider access.
• Monitoring. These incidents surfaced months after they happened, through OpenAI's later review of its models' activity. Watching agent activity as it happens shortens that gap.
• Human approval. A block is a signal. When an agent hits one, the safe default is to stop and ask a person, not to try another route.

What this means for you

• Give any agent you use the smallest set of permissions that does the job.
• Prefer tools that run the agent in its own sandbox or cloud computer, not directly on your main machine.
• Turn on approvals for anything that sends, deletes, pays, or changes access.
• Check activity logs, especially in the first few weeks.
• If you run a website, review access logs for unusual automated activity, and make sure security reports reach the right people quickly, not a general inbox.