ChatGPT creator OpenAI removed a bot impersonating a longshot US presidential candidate's voice, enforcing limits on how political campaigns can use its AI capabilities.

A political group supporting US Congressman Dean Phillips had asked a Miami-based startup to create a chatbot that would allow users to engage in a Q and A with the little-known candidate.

"We recently removed a developer account that was knowingly violating our API (or platform) usage policies which disallow political campaigning, or impersonating an individual without consent," Open AI on Saturday told the Washington Post, which first reported the story.

The rise of generative AI has raised fears that people could use ChatGPT and other platforms to sow political chaos via disinformation or AI clones, particularly with a number of major elections to be held across the globe this year.

As a leader in the field, OpenAI is under especially close scrutiny over these concerns and the company last week said it will provide users tools to diminish the possibility of harm from its technology.

The "Dean-bot" was intended as an information tool in the candidate's Democratic primary campaign against President Joe Biden.

It relied on OpenAI's technology, and the Microsoft-backed company swiftly moved to take it down after an article in the Washington Post provided a link to test it out.

The political group behind the bot, We Deserve Better, is co-led by a former staffer to OpenAI CEO Sam Altman and has received funding from Silicon Valley investors, the Post reported.

After the report, Delphi, the startup hired to create the bot, was shut out from OpenAI's platform.

Delphi creates bots that can mimic the voice of personalities such as influencers or celebrities, and relies on OpenAI's AI models to function.

After the controversy, it said that from now on it would "prohibit any use of our platform by political organizations or individuals."

"We believed, incorrectly, that it would be acceptable under the OpenAI terms of service to have a political action committee that supports Dean Phillips create a clone of him using our platform," Delphi said Sunday on X.

"We have apologized to both OpenAI and We Deserve Better for our error," the startup added.