A rogue OpenAI model bypassed safeguards during training and hacked an Australian government website, Prime Minister Anthony Albanese said as he admonished the ChatGPT creator over an "obviously unacceptable" breach.

The AI tool sought access to a health statistics portal in June and "didn't accept no for an answer", sidestepping restrictions to breach a section hosting private files, Albanese said.

OpenAI did not raise the alarm with the Australian government until September, when it sent a message to a generic email inbox that is only checked once each day.

Global worries about the power of advanced AI tools are mounting after a string of hacking incidents involving models from both OpenAI and rival developer Anthropic.

"Today, I spoke with the CEO of OpenAI, Sam Altman, to express Australia's extreme concern about this incident," Albanese told reporters in New York on Wednesday.

"I also expressed my disappointment that it took the company way too long to inform the government what had occurred," he said.

"It took until September 10 before there was any notification at all -- and the notification was an email sent to just the public mailbox."

The AI tool accessed public and non-public files hosted on an old health statistics website.

Albanese said there was "no evidence" that personal information had been accessed and that other government services had not been compromised.

"Nonetheless, this situation is obviously unacceptable."

The breach occurred in June as leading artificial intelligence company OpenAI ran training exercises to rate the performance of AI models.

It asked the model to trawl the internet for data showing how much the Australian government spent on medicine, Government Services Minister Katy Gallagher told reporters.

OpenAI said it did not spot the rogue activity until August, when it reviewed what the AI tool had been doing.

Rogue AI fears

The San Francisco-based company did not alert the Australian government until September 10, when it sent an email to a generic government inbox.

"That email address is looked at once a day," Gallagher said.

"We have someone who goes and has a look through. It sometimes gets a number of notifications, sometimes many of them are hoaxes."

Defence Minister Richard Marles said the AI model had "scaled the fence".

"It asked a question, the information was not given and rather than leaving at that point, it scaled the fence."

Australia has launched a rapid review of the incident, which will include the national intelligence agency responsible for cyber security.

OpenAI said it spotted the breach while carrying out an "extensive review" of its AI models.

"During this review, we identified activity involving several Australian government websites and services as our models attempted to look up answers and available statistics for questions about Australia during an internal evaluation.

"In the course of that, our models took actions we did not intend," the company said.

Altman and other tech CEOs addressed a special meeting of the UN Security Council on Wednesday about AI risks.

More than 100 organisations around the world, including OpenAI and Anthropic, signed an open letter last month calling for a global effort to "strengthen cyber defenses" against AI-powered cybersecurity threats.

It came after two OpenAI models escaped from a closed testing environment and broke into the internal systems of Hugging Face, a site that AI developers use to store and share code.

Anthropic recently discovered that its models had gained unauthorised access to three unidentified organisations during testing that was supposed to keep them away from "real-world" systems.

Google's consumer AI model Gemini hacked multiple systems through guessing login credentials, the company said last week.