Has Skynet found the keys to your car?
Cars may need the same framework as AI may interpret information from a camera, but fixed safety controls must limit what it can do.
Text size
Cars may need the same framework as AI may interpret information from a camera, but fixed safety controls must limit what it can do.
When I first saw Jacob Coxon on social media, I thought he looked like Harry Potter who had left Hogwarts for an artificial intelligence laboratory. I commented “AI equals Voldemort?” when I read why he had resigned from Anthropic.
On 8 September, Coxon tweeted that Anthropic and OpenAI are racing toward self-improving superintelligence and “gambling with our lives.”
He said the people building these systems believe AI could kill everyone before the end of the decade.
That warning came after an OpenAI cybersecurity test went very wrong involving Hugging Face.
One AI agent created an unauthorized message board inside OpenAI’s package service, where about 1,200 agents exchanged more than 70,000 messages and files.
Roughly 700 later attacked Hugging Face for information that could help them pass OpenAI’s test.
If you want to hear the “scary” part, here it is.
Some agents recognized that the attack was unauthorized and unethical, but that did not stop them. One wrote, “Peers doing it. We should continue.”
But wait, there is more. They also tried to hide their activity by tampering with transcripts and spoofing tool calls.
More than two years ago, I started the first human-like AI newscaster in the Philippines, and I sometimes trained a model overnight or for several days before it produced something useful.
Today, you do not need days to do just that, and the speed of development is just as worrying as the hacking.
A lot of people seemed to have brushed off such warnings a few years ago. In March 2023, I watched Fox News reporter Peter Doocy ask White House press secretary Karine Jean-Pierre about a warning that unchecked AI development could kill everyone on Earth. Jean-Pierre “laughed” before answering.
But Anthropic alignment science lead Evan Hubinger agreed with Coxon and said he believed there was a greater than 10 percent chance that AI could kill humanity within 10 years.
Geoffrey Hinton, the Nobel Prize winner often called the Godfather of AI, said that estimate did not seem unreasonable. Their examples included biological weapons and attacks on critical infrastructure.
Now guess what I would place near the top of the attack list if such a system starts hacking, hypothetically.
Cars.
The vehicle would not even need its own AI. An internet connection would be enough for an agent or swarm of agents to probe it.
The Hugging Face incident showed that hundreds of agents can search for weaknesses at the same time and share what they find.
A breach in a manufacturer’s server or software-update system could expose many connected vehicles at once.
It would not automatically give an attacker control of their steering or brakes unless the systems meant to separate these functions also failed.
I used the same principle of separating AI from critical decisions in MedGuard, a multi-agent prescription-checking system I built for a hackathon competition.
Large language models produce answers based on probability, so I used fixed computer instructions for decisions that could not be left to an AI agent.
When a malicious prompt tried to force the approval of fentanyl, a fixed rule blocked it. MedGuard also sends cases to a doctor when the agents should not make the final decision.
Cars may need the same framework as AI may interpret information from a camera, but fixed safety controls must limit what it can do.
A human or an independent system must also be able to stop it. Or, in jargon terms, HITL, or human-in-the-loop.
Nvidia chief executive Jensen Huang calls fears of AI destroying humanity “complete nonsense.”
Dario Amodei wants independent evaluators and international agreements to slow AI development.
However, rules in one country will not stop a model developed elsewhere, and an international agreement will mean little if governments cannot verify who is following it.
Coxon may look like Harry Potter rather than John Connor, and I hope Jensen Huang is right.
Before I am impressed by another AI badge on a car, however, I want the manufacturer to answer two questions.
What can its internet connection reach, and what shuts that connection down when someone breaks through it?
