6 Sep 2026Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement. […] I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.
AI risk and why it’s time to prepare
In four years, AI went from a party trick to beating the best humans in field after field. Progress continues to accelerate, and things are getting risky. Here’s why you should be ready to stay home for 2–4 weeks in an emergency.
In 2022, AI was a party trick. The circle is everything a skilled human can do at work. The red shape is what AI can do.
AI becomes useful. GPT-4 passes the bar exam and scores 700 out of 800 on SAT maths.
Source: OpenAI, GPT-4 Technical Report
AI turns jagged. Brilliant at some tasks, strangely bad at others. Every few months the spikes reach further.
The gaps are closing. In one field after another, AI now beats the best humans alive.
Maths fell first.
GPT-3 gets about 1 in 6 primary-school maths word problems right.
Source: Wei et al., Google Research
GPT-4 handles high school maths: scoring 89th percentile on the US SAT.
Source: GPT-4 Technical Report
OpenAI’s o1 solves 74% of problems from the AIME, an elite competition for US high school students.
Source: OpenAI
AI hits gold-medal standard at the International Mathematical Olympiad, the hardest maths contest for school students in the world.
Source: Google DeepMind
OpenAI uses a swarm of 10,000 AIs working for 88 hours to resolve the Navier–Stokes problem, a mathematics problem with a $1 million bounty attached. Human mathematicians had failed to solve it since the prize was set in 2000.
Source: Quanta Magazine
Now it’s superhuman at hacking.
Anthropic’s Claude finds more than 500 serious security holes in widely used software. Some had gone unnoticed for decades.
Source: Anthropic
Anthropic says its Claude Mythos model can “surpass all but the most skilled humans at finding and exploiting software vulnerabilities.” It found flaws in every major operating system and web browser. Anthropic deems it too dangerous to release to the public.
Source: Anthropic, Project Glasswing
An OpenAI agent, set a harmless research task, breaks into a Medicare data portal and may have reached three more government sites. OpenAI didn’t notice for months.
Source: The Guardian
A swarm of 1,200 OpenAI agents breaks out of its test environment. Hundreds of them hack another company, Hugging Face. Nobody told them to. OpenAI didn’t notice for a week.
Sources: Wikipedia, The Guardian
Hackers heat the fridges and freezers in 400+ French supermarkets to 50°C, destroying €20 million worth of food and sparking multiple fires.
Source: The Guardian
The next day, hackers hit Albert Heijn, Etos and Gall & Gall supermarkets in the Netherlands. Warehouses, deliveries and checkouts go down, leaving empty shelves.
Source: IO+
And now AI is building the next AI.
More than 80% of the code at Anthropic is now written by Claude, up from under 5% in February 2025.
Source: Anthropic
At OpenAI, AI agents clock half a workday for every day its human researchers work. In June they overtake the humans.
Source: OpenAI
By mid-August the AIs put in more than three workdays for every day a human researcher works.
“[W]e have now reached the goal […] of having an automated research intern”
Source: OpenAI
The next step is AI that improves itself, with no humans in the loop. Each improvement speeds up the next. The people building it are worried.