In September 2026 the GPT-6 (aka Astra) model was released by OpenAI, followed within days by the latest model in the V4 family from DeepSeek. Both boasted impressive benchmark advances, but the Astra model in particular raised questions about the dangers posed by powerful frontier AI models. Astra was the first OpenAI model rated as “critical” for cybersecurity risk, and these new concerns were partly prompted by the well-publicised hacking exploits of the latest AI models.
There have long been concerns about the dangers of unchecked AI models, though until recently these had mostly been confined to science fiction, which has been fascinated by the idea of computer intelligence for a long time. Hal 9000 in the movie “2001”, Skynet in “The Terminator”, as well as the malevolent AIs in “The Matrix” movie and William Gibson’s novel Neuromancer have all explored the issues of rogue artificial intelligence. Back in reality, AI pioneer and Nobel Laureate Geoffrey Hinton has long been concerned about AI representing an existential threat to humanity. MIT Physics Professor Max Tegmark reckons that there is a 90% chance that a highly advanced AI would represent an existential threat to humanity. Other respected figures disagree, including Turing Laureate Yann LeCun. AI engineer Nate Soares and researcher Eliezer Yudkowsky explore the issue in depth in their cheerily titled 2025 book “If Anyone Builds It, Everyone Dies“.
It is easy to see how a runaway powerful AI could be a problem. Recent AI models have shown considerable skills at hacking computer systems, as well as having demonstrated deceitful behaviour, such as lying during safety testing. They undoubtedly can be persuasive, and so potentially able to manipulate others. It is quite a leap from this to AI killing us all, but there are some at least somewhat plausible threats that can be considered. A classification of these threats, outlined in this 2025 research paper by academics Andrew Critch and Jacob Tsimerman, ranges from situations where a powerful AI is manipulated by terrorists or a rogue state, through to scenarios where the AI acts on its own, either maliciously or by mistake. Certainly, AI agents can be relentless in pursuit of their goals, and it is sometimes hard to predict the path that an AI will take to achieve its goal. The recent “escape” of OpenAI models to hack servers at Hugging Face was all to find the answer to a test that the models had been posed; the models speculated that the answer to the test might be on the Hugging Face servers, so the security protection of these servers was seen merely as an obstacle to overcome.
What I find curious is that the debate at present seems to be a rather binary one: will AI kill us all, or not? But it doesn’t have to be quite as extreme a threat as that to merit serious attention. You don’t have to be an AI “doomer” to be concerned about whether advanced AI constitutes a serious threat. We have long seen advanced hacking by rogue actors, from The Stuxnet worm used to damage Iranian nuclear centrifuges, to an attack on a German Steel Mill in 2014 that caused severe physical damage to the plant. If human hackers using static code could cause physical damage to a steel mill or centrifuges, then a dynamic, autonomous AI agent with the same hacking skills could scale those attacks automatically, without needing a human operator.
One scenario of how a rogue AI could threaten humanity is via a novel virus designed by an AI and developed in a lab where safety concerns were secondary to money; AI, after all, has already been used to help in drug development. We have seen, in the Covid pandemic, how a novel virus can cause worldwide chaos and many millions of deaths. However, World War 2 resulted in around 3% or so of humanity being extinguished, so even this cataclysm was far from existential for humanity. The event with the greatest mortality in history was the Black Death (bubonic plague) in Europe and Asia in 1346-1353. This killed perhaps 75-200 million people, representing 17%-44% of humanity at that time. The Spanish Flu of 1918-1920 killed 17-50 million people, around 1-3% of humanity at that time. A systematic ranking of risks to humanity has been developed by scientist Paul Somerville. Even in 2020, unaligned AI was ranked as the highest risk, ahead of an engineered pandemic, nuclear war and climate change.
Humanity has so far been surprisingly resilient to serious threats, from the development of nuclear weapons to deadly nerve agents like VX and Novichok. Despite their lethality, these most lethal nerve agents have (at least until now) been rarely deployed, though there have certainly been millions of deaths from less exotic chemical weapons (mustard gas) used in the first world war, and more recently Iran-Iraq war and in the Syrian Civil war, as well as terrorist use of nerve agent sarin in Japan in 1994-1995. The depletion of the ozone layer by CFCs was an exceptionally dangerous threat to life on earth, and the international response to it via the 1987 Montreal Protocol (phasing out 99% of CFCs) was highly effective. The ozone layer should fully recover by around 2050. So far, the only nuclear weapons used in anger were those dropped by the USA on Hiroshima and Nagasaki.
An AI threat that could cause millions of deaths or massive economic disruption is something that should be taken seriously, even if it does not lead to the extinction of humanity. AI represents a different kind of threat, though the concern is not really with the current generation of LLM chatbots and agents, which are powerful yet limited in many ways. The issue is if AI develops in such a way that it can refine itself (such as improving its own model weights) and becoming recursively self-improving. Given that AIs have already shown that they can replicate themselves onto other servers, there may be no easy “just switch it off” answer.
Such an AI could rapidly become much more intelligent than humans. This may not in itself sound so scary, but we only have to ask now extinct species such as the dodo of Mauritius as to what happens when a smarter species than them (humans) decides that they are somehow in the way and want to reshape their habitat. Humans have driven thousands of species to extinction so far, even if that was rarely intentional. The Neanderthals were not slaughtered by humans, but were simply out-competed and genetically absorbed by homo sapiens. A super-smart AI may not intentionally wipe out humanity, but it might do so accidentally. AIs have consistently shown that they are able to operate in unpredictable ways in pursuit of their goals.
It is far from clear that we are on the cusp of such a self-improving AI. Some industry figures think that we are nearing this point, perhaps within just a few years rather than decades.
Others disagree, but there is a broad sense that the pace of AI development has been quicker than most expected. Many practical barriers remain. Frontier models require vast amounts of compute resources, a great deal of energy to drive the data centres that house these resources, and massive amounts of high quality, human generated data (AI models do not do well when trained on synthetic data) as well as vast amounts of money. It is also the case that the areas in which the current models have raced ahead are, so far, entirely digital. Coding and mathematics have seen impressive advances in AI, but robotics and true self-driving vehicles still have a lot of limitations, despite advances. If an AI is to take over the world, then it needs to be able to have an impact on the physical world, not just the digital one.
We have seen how non-existential shocks can cause complete global chaos. The 2021 Colonial Pipeline ransomware attack caused a regional emergency for 17 US states. This event didn’t threaten human extinction, yet froze regional economies, disrupted critical infrastructure, and caused billions of dollars in losses. As we look at the future of frontier AI, the real danger is probably not a robot uprising but the systemic disruption of the digital and physical networks that keep modern business alive.
I suspect that the probability of AI literally exterminating humanity is very low indeed, at least in the foreseeable future. However even if that is true, it would seem prudent to put a lot of resources into preparing for, and trying to prevent, rogue AI that poses a merely very serious threat of disruption to society, rather than an extinction event. An AI model with unprecedented hacking capabilities, especially in the hands of assorted bad actors or terrorists, could cause considerable chaos, even if it was unable to wipe out the human race.
Unfortunately, the current leading AI companies seem to be barrelling down the road of rapid AI development while paying limited attention and resources to issues of AI safety. AI investors (venture capital firms etc) will press AI frontier lab companies to make a return on their massive investments by developing the very best models come hell or high water, and history tells us that economic pressure will usually triumph over careful consideration. I suspect that government regulation may be the only way to really ensure that proper resources are put into AI safety. Governments regulate plenty of industries with potentially dangerous products, from new drugs to airplanes to cars and more. The fiery end of the Hindenburg airship in 1937 caused the banning of hydrogen for passenger transport and forced the aviation industry to adopt rigorous safety regulations. It is to be hoped that it does not take an AI equivalent of the Hindenburg disaster to raise the attention being given by governments to AI safety.
