POD! On YIMBY Pod this week, we talk about the government’s latest efforts to make it easier to build near train stations and densify our cities. Then we speak to Nick Maini about why Hammersmith Bridge has been closed to public transport for six years – and how autonomous pods could fix it. Listen here, or wherever you get your pods.
I swear I don’t mean to keep writing about AI, but it’s the biggest story in the world, and I think this is particularly important…
Every so often, our lives are punctuated by shocking events that change everything.
9/11 is an obvious example. I was 14 when it happened, and though I didn’t fully understand it at the time, I still remember having the visceral feeling in the pit of my stomach that nothing was ever going to be quite the same again.
I felt something similar a few weeks ago. I was watching two OpenAI employees give a presentation at the DefCon security conference, detailing what happened in the now-infamous cyberattack that OpenAI’s AI models autonomously carried out against Hugging Face, another AI company.
In case you missed it, the technical details are complicated but the story is simple. An AI model was being evaluated internally. It had essentially been tasked with an assignment to solve a particular digital security puzzle. But due to an error on the part of the testers, the AI was unable to access some of the documents it was supposed to use to find the solution.1
However, instead of giving up and declaring the task not possible, something extraordinary happened. Over a period of a couple of months, the AI explored alternative solutions to the problem it had been tasked with, and essentially discovered that it could create a 'message board’ where it could interact with other AI models inside the company. Then astonishingly, unbeknownst to any human, the AIs figured out how to work together to not just break containment and access the internet – but they reasoned to themselves that the best way to solve their problem was to hack Hugging Face. Which they did successfully.2
I’m assuming that after this, the examiners were not too worried about the examination, as this whole incident was an absolutely spectacular demonstration of just how sophisticated AI has become in such a short space of time. Give a sufficiently powerful model a task and it will pursue its goal relentlessly until it finds a solution.
This brings me to Britain.
Though we are only a middle power, and the ‘frontier’ models like ChatGPT and Claude are being developed elsewhere, we have so far carved out a rather successful niche in the AI present. The AI Security Institute (AISI) was created by Rishi Sunak, and in less than three years has proven itself seriously credible in the eyes of both the tech industry and foreign governments.
AISI has assembled a team of world-leading experts who, when new AI models are created, can credibly evaluate the threats they may pose. When Claude Mythos, a powerful new model, was created by Anthropic earlier this year, AISI was granted early access to test the model – and it published a warning about its capabilities, which was taken very seriously.
However, as effective as this new institution is, following the Hugging Face incident I’m not convinced it is enough if we want to protect Britain and our allies from AI threats in the future. 9/11 was a clear indication that global peace and stability were about to be upended, and Hugging Face is the same. And unless Britain acts now, it won’t be ready for the threat that is rapidly emerging.
So this week, please allow me to explain the emerging problem – and what we can do about it.
I’ve got so annoyed about the discourse on this, I’ve now removed the paywall on this post. So if you find it useful or informative, please consider subscribing.
The new security threat
As Ben Thompson has observed, the next few years are going to be a security nightmare.
However optimistic I may be about the potential good things that AI will make possible, there is going to be an awkward period of adjustment as sophisticated AI models pick over the code that humans have written over the last half-century. AI will reveal bugs and vulnerabilities at an unprecedented scale – in software that we use every day, and in code that underlies the infrastructure of the modern world.
This isn’t because the code is bad, per se. It is because writing code is inherently an imperfect process. An operating system like Windows or MacOS, for example, is the combined work of tens of thousands of people. It is so bewilderingly vast and complicated, that it simply isn’t possible to ensure that it is impervious to threats.
Think of it like an airport. We can add layers of checkpoints, bag searches and x-ray machines to reduce the security risks, but it is impossible to make such a complicated building completely safe.
So since the dawn of computing, cybersecurity has been a permanent fact of life. The bad guys find vulnerabilities, and the good guys update the code to make it safe again. This is a never-ending process, and it is the reason why with annoying regularity Windows will insist on installing updates when you power down, or MacOS will randomly reboot to update.
For a long time, the game of cat and mouse has just about held – but now AI is getting involved in finding vulnerabilities. And we’ve already seen what that looks like – it’s overwhelming.
When Claude created Mythos, it let the makers of some of the world’s most critical software use the new model to look for vulnerabilities in their own code and the results were striking. Cloudflare, which provides important internet infrastructure, found over 2,000 bugs (including 400 critical security issues), and Mozilla, which makes the Firefox web browser, found 271. And in addition, over 3,900 high or critical security vulnerabilities were found in an analysis of various open source projects.
And it is not just Mythos that is the problem. Earlier this summer, OpenAI used its latest model, GPT-5.6 to find numerous bugs in apps like Chrome and Safari, as well as in the kernels of Linux and FreeBSD3 – two operating systems that run the vast majority of the server infrastructure across the internet.
Pandora’s box
Perhaps you can see the problem. We’ve now got highly sophisticated AI tools that can detect vulnerabilities at unprecedented scale, far faster than they can be patched by traditional methods – and they will relentlessly pursue the goals they have been given. This is not a hypothetical future – this is the world we live in today. And right now is the worst they will ever be.
In the early days of the post-ChatGPT AI explosion, some serious people talked about how governments could set rules to slow or regulate AI progress. This was basically one of the reasons the AI Security Institute was created in the first place – to give the government a role in deploying this potentially dangerous new technology.
But now it is clear that the brakes cannot be applied.
If we lived in a world where OpenAI and Anthropic were the only companies that had this technology, it might conceivably be possible. But unlike other dangers like, say, nuclear or biological weapons, we can’t control AI models with treaties or restrictions on materials. The technology runs on commodity computer hardware and the models are ultimately just large text files full of numbers.
And even if we could, somehow, control these things, then there’s the problem that other countries are working on AI too, including most notably, China. And this makes clear why we now face an unprecedented security threat.
So far, China’s AI progress has lagged the US, but it has consistently only been behind by months. When it was first released last year, DeepSeek caused a minor freakout as it was almost as sophisticated as the best ChatGPT model available at the time, and shockingly it ran much more efficiently. Then more recently, another Chinese model, Kimi K3 has done the same. Though comparing AI models like-for-like is difficult, on some benchmarks it is only narrowly trailing the capabilities of GPT-5.6 Sol and Claude Fable.
This is to say that the genie on this technology is already out of the bottle.
Perhaps if Chinese companies had pursued an approach to development similar to OpenAI and Anthropic, it might be possible to just about imagine a modern equivalent to an arms control treaty between the major powers. Maybe.
But the problem with this though is that China is pursuing a completely different AI strategy, and its government and businesses have wildly different incentives.
As China is currently trailing US capability, most Chinese models are open source and/or open weight – meaning the raw code and data used to create the models have been published freely for anyone to use.4
Why would Chinese companies do this, given that Anthropic and OpenAI have both received astronomical valuations in part by keeping their models closed? Isn’t that the ‘secret sauce’ that makes the companies so valuable? The difference is that because China is trailing on raw capability, it has decided to compete instead in the ecosystem around the models. (This is a strategy that has come all the way from the top, with Xi Jinping himself calling for this.)
So the hope is that people and companies will choose to use Chinese models because open models can work out cheaper to run. Openness also creates an incentive across the companies that use them to work out how to run them even more efficiently, reducing the need for cutting-edge Nvidia AI chips, which are subject to export controls that have been imposed on China by the West.
And though this is great from an AI-ecosystem perspective, arguably for everyone in the world except Anthropic and OpenAI, openness does also have one difficult geopolitical consequence: As anyone can use them, it really does mean that anyone can use them.
That could mean that rival powers like China and Russia, rogue nations like Iran and North Korea, or hackers and terrorist groups. They can take these open models and do with them, well, whatever they like. Unlike closed models, where usage runs through OpenAI and Anthropic’s servers, if you have enough compute capability you can run these models without restrictions on nefarious uses.5
So this is why I think we’re hurtling towards an AI security crisis. Though the open-weight models are still not as capable as GPT-5.6 Sol, or Claude Fable, if history is any guide they are not far behind. And when they catch up, there will be no effective limitations on their use – we will have to assume that the worst possible person or group, with the worst intentions, will make use of these models too.
And that’s why we need to prepare for that world now.
A war of resources
Let’s go back to Britain.
Choose your own ominous metaphor: It’s 1938 and Germany is rapidly rearming. It’s 2001 and a CIA memo is warning that Bin Laden is determined to strike the United States. It’s 2013 and Vince Cable is signing the paperwork to sell off the Postcode Address File. It’s 2020 and there are reports of a strange virus in Wuhan.
In the not-too-distant future, it is incredibly likely that we will be living in a world where sophisticated AI models, which cannot be controlled or regulated out of existence, can be deployed to relentlessly and autonomously assault every digital system in our lives by actors unknown.6
How does global power work in this world? And how can Britain protect itself?
In terms of the mechanics of defending, the old paradigms of cybersecurity are dead. Any human-led efforts to patch and update software will simply move too slowly. They will not be able to keep up with the AI assault.
So the weapon we will need to use to defend ourselves is, well, AI. Just as AI can find vulnerabilities at speed, it can also conceivably patch them just as quickly too.
In other words, the only way to stop a bad guy with an AI, is a good guy with an AI.
If you’re a traditional IT security person reading, this might sound like a heresy. What I’m arguing here is that it simply won’t be possible to have a human in the loop, reviewing code before it is patched in. We will need to defer authority to defensive AI systems to update critical systems to defend them. There will be no other way to do this.
And the consequence of this fact is something quite profound, I think, as it essentially reshapes global power.
Power has always been a function of hardware. The number of tanks, missiles and (increasingly) drones that a country can deploy defines its ability to defend itself.
But in a post-Hugging Face world, the hardware we need has changed again. Simply put, power in the new world will be a function of how much compute a nation has at its disposal. If every digital system is under constant attack by autonomous AI agents, we will need to have the capability to stand up to such an assault.
And, sadly, I’m just not convinced that Britain is appropriately readying itself for the battle ahead.
Protecting British interests
So how is the British government supposed to respond to the new reality?
If we want to put ourselves in a strong strategic position, there are essentially three core interventions we could make in the AI ‘stack’: on models, compute, and energy.
On models, if we had our own frontier models, it would put us in a strong position. Our AI could literally outsmart the other AI. This is how we get away with having relatively few soldiers – we rely on our missiles and our planes being better than those of our enemies.
But sadly, in the case of AI we do not have our own frontier models – and developing one would be extremely difficult.
And it’s worth dwelling on this to understand exactly how difficult it is. We’re not just one extra commitment casually thrown out by Andy Burnham away from making it happen, as catching up with the best American and Chinese models would be almost impossibly expensive.
For example, Meta – which is desperate to catch up with OpenAI and Anthropic – has reportedly already sunk over £100bn this year alone into its ‘superintelligence’ lab, and it still appears to be nowhere close to developing a similarly top-tier model. And for context, that’s equivalent to around half of the NHS’s annual operating budget. So in terms of public spending, building our own model is a non-starter for obvious political reasons.
Instead, our only option model-wise is to hope that, despite the Trump of it all, the United States continues to let Britain access the most sophisticated models. This is why having AISI is such a useful bargaining chip as America likes its credibility too.
However, assuming we have a sophisticated model, we need more than this too – we need the physical hardware to run it on. And this is where I do think we can do something useful.
If power is a function of compute, then we can simply build more data centres, and generate more energy with which to power those data centres. The more AI tokens we can crunch, the stronger our defences will be against the threats to come.
And to an extent, this is something the government is already working on. There’s no indication that Andy Burnham intends to change the AI Growth Zones plan, and there’s still a pipeline of renewables coming online (along with Sizewell C and Hinkley Point C). So in due course, we’ll certainly have more data centres and energy generation to power them.
But my concern is that given the new security paradigm I describe above, this still isn’t fast or large enough.
For a start, there’s a huge misalignment in terms of timelines – data centres and power plants currently take years to build, and yet AI progress, and the lag between the frontier and Chinese open source models can be measured in months. It will be surprising if China doesn’t have a model as powerful as GPT-5.6 or Mythos by Christmas.
And this wouldn’t be so bad if AI was just an economic nice-to-have. If we don’t have the latest technology the fastest, it merely means sluggish economic growth and a loss of competitiveness to our peers.7
But viewed from a national security perspective? Suddenly that capability gap feels a lot more urgent.
However, there is one reason for optimism here. As urgent as the need to build our AI infrastructure is, it need not be costly to the public purse. Unlike building a frontier model, which would require enormous upfront capital spending, data centres and new sources of electricity are economically useful in their own right. They can both help the economy and act in a dual-use capacity for our cyber-defences, should we require it.
So this means that new data centres can pay for themselves. In fact, given the AI boom, the private sector is pretty desperate to build out capacity. All the government needs to do is, well, let them build.
And this is why, finally, I want to end by proposing a policy that is simultaneously deeply important if we want to protect Britain’s national security, while also perhaps the single most electorally unpopular idea anyone has ever proposed.
I’m proposing that the government needs to create ‘Nightingale’ data centres.
These are, of course, named with reference to the conference centres full of hospital beds that were spun up in the early days of the pandemic. What’s remarkable wasn’t that they were not actually all that useful (we didn’t know that at the time) – what matters is that we built them quickly.8
AI data centres are more technically demanding than a room with hospital beds. And I’m not proposing we start setting up server racks in the ExCeL Centre. But we should take that same urgent spirit and apply it to the data centres we’ll need in a new world that has arguably already emerged.
At a minimum, plans for data centres and energy infrastructure should be rapidly called-in by the government and even more rapidly approved. Process should be vastly curtailed, and every official should be driven by the need for rapid delivery. Like during Covid, we correctly decided that now wasn’t the time for layers of consultation and review – the need for compute should be approached the same way.
I still believe that, ultimately, AI will do incredible things to improve our lives. But the Hugging Face incident was a warning, like the Bin Laden memo sent a few weeks before 9/11, and we should act now to avoid the inevitable security crisis once the Chinese models catch up.
Digital people, please forgive the mild simplification here as I want to make the story legible for a general audience, without having to get into a long explanation of what a package manager is.
It’s funny looking back on the ‘Moltbook’ controversy now. That was a Reddit-style message-board setup for AI agents to interact, and it quickly showed the ‘agents’ creating their own threads with in-jokes, and shitposts. There was a huge split between people who thought it was a real and interesting demonstration of something important, and people who thought it was fake and just humans posting pretending to be agents (there was little to no verification). I guess it turns out the most bullish AI people were right once again.
The kernel is like the foundational, base-level code that tells a computer how to interact with hardware – like how to use the processor or talk to the network ports. So pretty damn important!
An amusing consequence of this is that because everything is published openly, it’s possible to see where the censorship about things like Tiananmen Square is introduced. In most cases, I think training data isn’t censored, but the developers will add instructions to tune the model after it has been created. And then (as in the original DeepSeek release), the ‘harness’ – the website or API access to the model will also be where the censorship kicks in too.
A ‘fun’, non-hacking example of this was posted by Jeffrey Emanuel, who was testing the latest version of Qwen. This is a small model, that can run on a regular home computer (albeit a fairly powerful one). It didn’t blink when he asked how to make methamphetamine.
The battle may have already begun. It’d frankly be almost geopolitical malpractice from the perspective of the American security state if it has not deployed Claude Mythos to poke around in every Russian and Chinese computer system it can find, before they develop similar capabilities to stop them.
I mean, at least we’re used to slow growth and low productivity at this point, so it wouldn’t be anything new.
I’m sure some smart-arse might quip that similarly perhaps my belief we need more data centres might turn out to not be the case. Maybe so, but do you want to take that risk?
I don't understand why physically hosting the compute in the UK is so important, especially if we are going to be using foreign models that are available to our adverseries. I would rather focus on using the extra power required to shift away from fossil fuels for heating and transport.
We will probably suffer more cyber security attacks, but unless we have access to the source code of the vulnerable software then we won't be able to patch it. If we do have access, then its likely to be open source so someone else will be able to patch it.
Maybe the money could be spent on sponsoring open source projects to help keep them secure and to move government departments over to said open source and away from the closed source systems that would be more vulnerable.
In general I agree that people are not freaking out enough about the Hugging Face hack, and more generally about how good AI has got at cybersecurity. But the current deluge of vulnerability reports probably won't continue at the same rate forever. The usual pattern with any new bug-scanning tool is that it finds a huge number of bugs the first time you run it against a new codebase, but once those bugs are fixed the rate drops off. As the tool improves it finds a few more bugs, but all the easy pickings were found in the first run. Then you try another tool that works on different lines to anything you've tried before, and the cycle begins again...