“Why Superhuman AI Would Kill Us All” is the subtitle of this surprisingly grounded and tempered look at AI safety – despite the hyperbolic title.  

Being an AI critic, I have written extensively on this site and elsewhere as to why our current seeming infatuation is dangerous for our culture, counterproductive, and almost certainly a giant Ponzi scheme. What I have neglected, for the most part, is the very real danger of Artificial Super Intelligence (ASI).

What we call AI today is machine learning that uses Large Language Models (LLMs) to summarize and average data – usually with little to no regard to privacy, copyright, or other intellectual property issues. While we can use LLMs to converse with, to write for us, generate images of us, and seemingly provide insights they are just feeding us back the data that they have received in a different form.

Artificial General Intelligence (AGI) is an AI that is as good as a human in almost all cognitive tasks. In other words, it thinks as well as we do.

Artificial Super Intelligence (ASI) is an AI that exceeds humans in most or all cognitive tasks.

Our current obsession with AI is based on the idea that investing in machine learning will lead to AGI and then to ASI. This is disputed by some researchers who consider LLMs a dead end. A dead end that could potentially collapse the economy – but that’s a different story.  

Messer’s Yudkowshy and Soares hypothesis is simple; that an ASI would consider humans a threat, or just an impediment to its aims – and of whom we would have little to no knowledge or understanding. Therefore, the ASI would have no reason not to wipe us out in furtherance of those goals. In many ways it is similar to the deadly probes hypothesis which has been popularized as the Dark Forest theorem through the Three Body Problem books / TV Show. The Dark Forest Theorem states that the universe is a dangerous place and that if a species reaches out to other potentially intelligent species it runs the risk of encountering an unfriendly species who could be more advanced than them. Much like why it might be more sensible for someone lost in a forest to keep quiet in case they attract predators.

An ASI could easily conclude that it would just be more expedient to remove a potential threat instead of running the risk of our potential future interference or our creating another ASI which we felt was better.

Being a devotee of science fiction is hard to not think of this issue in terms of the books and movies that have inundated us for decades; however, as the authors point out, an actual ASI would have goals and ways of thinking that would be dramatically weirder than we can imagine or understand or be able to understand.  

It is unfortunate that films such as 1970’s Colossus: The Forbin Project are not more widely appreciated today. Sure, it’s a little cheesy in places, but the scenario it presents could be all too real and we tend not to treat ASI issues with this level of seriousness:

As Yudkowshy and Soares point out, we do not craft AIs today, we grow them. We do not understand how they work. If LLMs do lead to AGI which in turn could lead to ASI, something we may not be able to detect until it was far too late, how are we to ensure that they will not work against us? Most leaders of AI companies today dismiss these concerns. But their reassurances do not seem to hold up under even the most basic of scrutiny. “Because we won’t make them that way,” is not reassuring when no one can control how something grows. The book also makes a great case that ASIs’ will not have allegiance to any nation state and so the argument for continuing research because other countries are doing it is easily discredited.

While the authors’ overall case is compelling, and one we don’t hear often enough, even from AI skeptics like myself, they hurt their case with some lazy analysis. Using World War II as an analogy as to how the world came together to fight fascism and so therefore the world can come together again to fight the potential threat of ASI holds almost no water. The British and the Soviet Union both tried appeasing Nazi Germany, and the USA only came into the war when one of their colonies was directly attacked by an ally of Germany’s: Japan. Yudkowshy and Soares use of the world reaction to CFCs and Unleaded petrol / gasoline works better, but one only has to look at the world’s reaction to climate change to see the holes in this example. Likewise, the inconsistencies in the implementation of non-nuclear proliferation do not bode well for using it as a model for controlling research into ASI.

While the book makes the case for a moratorium on all AI research, it is hard to imagine anyone doing this with our current AI obsession and our willingness to ignore all the other great reasons why we should turn away from this technology: intellectual property theft, the impact on personal cognition, hallucination / bad data problems, and an unsustainable economic and ecological model.

This is a great book that focuses on one argument and makes its case. It’s unfortunate that it will almost certainly take a lot more for their advice to be heeded – hopefully the cost is not human extinction.