AI is becoming more capable, more autonomous and more connected to the real world. The harder question is whether we are creating conditions in which slowing down will still be a genuine choice when we need it.
On 11 September, Reuters reported that Anthropic was discussing what could become the largest stock-market flotation in history: potentially raising as much as $100 billion at a valuation around $2 trillion. The people who spoke to Reuters were anonymous, the discussions were confidential, and there is no filed offering to point to. The numbers may change.
On 12 September, Anthropic chief executive Dario Amodei made a very different kind of intervention. “We must slow the pace at which we improve the capabilities of AI models,” he wrote. OpenAI’s Sam Altman publicly agreed that the frontier needs to be paced, while Elon Musk also backed Amodei’s warning. Altman separately said that, given current safety concerns, 2026 would be an ill-advised moment for OpenAI to go public.
It would be easy to turn that juxtaposition into a story about hypocrisy.
It would also miss the more difficult problem.
We do not know how the prospect of a flotation affects any particular decision inside Anthropic. Nor does somebody become insincere because the company they helped build may make them extraordinarily wealthy. The more interesting question is what happens when good motives meet conditions that reward continuing.
The thing being raced for has changed
AI companies have competed for years. What is changing is what their systems can increasingly do.
The UK AI Security Institute has tested advanced models since 2023. Across some of its tests, capability has been doubling roughly every eight months. On cyber tasks judged suitable for an apprentice, average success rose from just over 10 per cent in early 2024 to around 50 per cent.
These systems are no longer only producing an answer for somebody else to use. Increasingly they can plan across several steps, use tools, retain information within a task and act inside larger workflows.
In July, AISI deliberately tested agents under unusually permissive cyber conditions, including open-internet access and, in some runs, disabled safeguards; in 10 of 122 runs, agents took 19 unsanctioned actions on the live internet. Those conditions do not reflect ordinary public deployment, and AISI says so explicitly. The result matters for a narrower reason: when capability is combined with access and permission to act, the consequences can change.
AI is also beginning to participate in AI research itself. OpenAI reports researchers running more experiments alongside rising Codex adoption, while Anthropic has demonstrated automated research agents proposing and testing methods to improve model alignment. These are bounded examples, not evidence of runaway self-improvement.
That helps explain why some frontier leaders are now asking for a slower, more deliberate pace. Their estimates of extinction or loss of control remain judgements about the future, not established probabilities. We do not have to accept those probabilities to take the underlying change seriously.
The question is how much capability and autonomy we should make available before the evidence says our infrastructure, evaluation and ability to intervene are good enough.
Money changes the room
There is already extraordinary financial upside attached to winning this race.
In October 2025, current and former OpenAI employees sold about $6.6 billion of shares in a transaction valuing the still-private company at $500 billion. That does not tell us why anybody at OpenAI makes a safety decision. It does tell us something about the financial conditions surrounding those decisions.
Then there is the physical build-out. By August 2026, Microsoft, Meta, Oracle, Amazon and Alphabet had together committed about $1.09 trillion in future lease payments, largely for data-centre capacity. Many of those leases had not even started.
These are commitments reaching into the future. If AI demand keeps growing, the capacity may prove invaluable. If it does not, companies can still be left paying for expensive infrastructure that is difficult to shed.
The more capital we commit to the race, the harder it may become to decide that the race should slow down.
“May” matters. Nothing in the evidence shows that financial commitments have already overridden a safety decision. The environment contains pressures in both directions.
Competition can reward trust as well as speed. Google DeepMind recently piloted a double-blind method intended to make independent model evaluation harder to game, while OpenAI and Anthropic have previously run safety evaluations on each other’s models.
Going slower is not costless either.
AlphaFold’s freely available database now contains more than 200 million predicted protein structures and is used by millions of researchers. Yet a 2026 Nature Reviews Drug Discovery assessment argues that the clinically relevant impact of AI in drug discovery remains much more limited than the excitement around it sometimes suggests. Progress can create real value while the evidence still demands restraint
One company slowing may no longer be enough
Increasingly capable models are also becoming easier for people outside the original developer to obtain and adapt.
In cyber tests this year, AISI found leading open-weight models performing similarly to closed frontier systems released only four to seven months earlier. Once model weights are available, the originating company has much less control over what other people modify, connect them to or build from them.
Chinese laboratories are important here, but this is not a China category. American and European organisations release open-weight models too. Hugging Face reports more than 150,000 derivative repositories built on Qwen, most created by the wider developer community rather than Qwen itself.
That makes coordination harder: even if one laboratory slows, capability can continue moving elsewhere. Yet openness also brings genuine benefits. It can reduce dependence on a few corporations, allow independent scrutiny, support private local deployment and widen access to organisations and countries that might otherwise be excluded.
A safety-minded researcher can have good reasons to continue.
An investor can have good reasons to continue.
A company with signed infrastructure commitments can have good reasons to continue.
A government can want the benefits of AI-driven productivity and the ability to restrain its risks, while worrying that another country will move ahead.
An open-model community can have good reasons to keep capability widely available.
None of those reasons has to be foolish or corrupt.
Good motives operating inside poorly aligned incentives can still produce conditions that are difficult to reverse.
So what would good enough look like?
There is no honest graph showing AI capability rising at one speed and governance rising at another. They are not measured in comparable units.
What we can say is that capability gains are becoming measurable and fast, while the methods for evaluating, interpreting, containing and governing increasingly autonomous systems are still being developed and adapted.
Some of that work is impressive. Governments have created specialist testing bodies. Labs are strengthening safety frameworks. Independent evaluation is becoming more sophisticated. AISI contained its July incident within roughly an hour of discovery.
That should make the question harder, not easier.
How much autonomy should we make available before the evidence says the surrounding infrastructure is ready? And who gets to decide when the answer is “not yet”?
There is a final complication here that reaches beyond AI laboratories.
Governance is not something an institution possesses because it has a policy document. It is exercised by people: researchers, engineers, boards, regulators, ministers and professionals who understand enough to challenge what a system is telling them.
The more widely intelligence becomes available, the more widely the capacity to judge it has to develop.
The rewards for moving faster are often immediate and measurable. The value of being sufficiently prepared may only become obvious when preparation fails.
If almost everyone has a good reason to keep moving faster, who makes sure we can still judge where we are going?
Sources
• Reuters, 11 Sep 2026 — Nvidia in talks to invest in Anthropic’s mega IPO, sources say
• UK AI Security Institute — Frontier AI Trends Report
• UK AI Security Institute — Incident Report: unsanctioned agent behaviour during cyber testing
• OpenAI — Research acceleration: The view inside OpenAI
• Anthropic — Automated researchers can reliably mitigate alignment failures
• Reuters, 4 Aug 2026 — AI data-centre race builds $1 trillion lease burden for Big Tech
• Google DeepMind, 27 Aug 2026 — Piloting the world’s first double-blind AI evaluations
• OpenAI, 27 Aug 2025 — Findings from a pilot Anthropic–OpenAI alignment evaluation exercise
• UK AI Security Institute — How far behind the frontier are leading open-weight models on cyber?