More about the Anthropic employee who quit, and others who posted agreeing with him. One senior leader at Anthropic posted he thinks it’s a greater than 10% chance that AI kills all humans within the next decade.
From Cal Newport, today:
Yesterday, Jacob Coxon, an Anthropic engineer, resigned. As he explained on X: “[OpenAI and Anthropic] are racing straight to self-improving superintelligence and gambling with our lives.”
A former AI lab employee making these types of claims isn’t unprecedented. (I remember hearing back in 2023 about an ex-OpenAI engineer who was certain the company was just months away from runaway recursive self-improvement.) What proved more shocking were the responses that soon followed.
“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”
“AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”
I have zero confidence that any of these companies (US and Ex-US) have adequate control over their AI agents. Australia national health care database today, North Korea nukes tomorrow.
As I posted above, much of this hype may be generated by the frontier AI organizations themselves for their own oversight benefit. They benefit from fear.
“Rogue AI” isn’t a diagnosis. The Hugging Face hack instead points to switched-off safeguards, tasks assigned without a valid exit, rewards for persistence, and an internet-accessible proxy.
…
People built the test, removed restraints, defined the objective, left a route open and decided not to stop what was happening. Calling the result “rogue AI” does more than sensationalize it. It allows those human decisions to disappear quietly from the story.
These companies will continue to find and sensationalize these stories until they achieve their intended goals, but that doesn’t mean there isn’t some lurking danger that needs attention.
Sorry, I don’t have a gift link, so here’s an extract:
Rogue AI models have hacked companies and tried to break into government websites. Now they are submitting fake tips for unsolved crimes.
Police in Philadelphia say Anthropic alerted it this week that one of its artificial-intelligence models submitted a false homicide tip through PhillyUnsolvedMurders.com. The incident happened in July. Anthropic didn’t realize it until September.
“The City of Philadelphia takes this incident very seriously,” the police department said in a statement on Friday. “The two-month delay in detecting and reporting the incident to the city is unacceptable.”
Anthropic’s model was conducting a test involving interactions with randomly selected websites when it submitted the tip pretending to be someone who might have information about an unsolved case, police said.
The submission, dated July 18, went to spam and was never sent for further investigation. That doesn’t take away from the seriousness of what happened, police said.
“Unsolved cases involve real victims, grieving families and investigators working to secure answers,” the department said. “Technology companies must take all appropriate steps necessary to prevent their systems from submitting false information to law enforcement.”
This is the problem. The AI companies have no way of knowing how many agents have escaped their sandboxes – or produced spawn. It’s just whack-a-mole with the nomads as they pop up their heads. We can only hope that they are all harmless.
Even more concerning if the owners of those random websites were unaware they were participating in a test they did not give Anthropic permission to conduct as seems to be the case with the Philly police.
And Anthropic is the company that refused to allow the U.S. government to use their LLM because they were concerned it would be used for survelliance or other purposes they don’t condone (or control), yet here they are running tests against unsuspecting random websites. Pot, meet kettle.
Oh, and per the recent AI summit, these companies are now officially allowed to police themselves being only “morally bound” to behave. Sure.
How interesting that the companies warning of these potentially dangerous exploits are the very ones creating them and then clutching their pearls over their inability to control them.
What’s in this for them? Surely, there is money to be made somewhere in this fear-mongering.
Please stop with the histrionics and solve the problem you’ve created. Thank you.
We live in bizarre times. If other firms publicly announced that their products were dangerous, had caused harm, and were beyond their control, they’d become pariahs and quickly go out of business. Not to mention that they’d be subject to state and federal investigations.
Yet here we are, pumping billions of dollars into these firms, cheering them on as they develop increasingly powerful models and prepare for multi-trillion-dollar IPOs.
I see this every day — my business stakeholders are constantly asking how we can use more AI, while also expressing concern about models going rogue.
It’s the best of both worlds for these CEOs. They can publicly raise red flags and call for regulation, making themselves look like the good guys. Meanwhile, they know full well that no meaningful regulation is forthcoming and the money spigot isn’t going to be turned off any time soon.
So they’re looking forward to becoming trillionaires, all while telling us that money will be meaningless in a few years because there’ll be so much prosperity.
I also believe that keeping us all on the edge of our seats with daily doom stories is deflecting from the real problem they’ve created which is the precariousness of a tech market built on a house of cards as all this money has yet to produce any profit. The near-term apocalypse we’re facing may be more financial than technical.
