Why I'm not afraid of superintelligent AI taking over the world

The world isn't a giant chess game.

I’m a journalist with a computer science master’s degree. In the past I’ve written for the Washington Post, Ars Technica, and other publications.

Understanding AI explores how AI works and how it’s changing the world. If you like this article, please sign up to have future articles sent straight to your inbox.

---

In 1997, an IBM computer called Deep Blue stunned the world by defeating the world chess champion, Garry Kasparov. Then chess software kept getting better. Today, the highest-ranked chess software, Stockfish, has an Elo rating of 3545, while the top human player, Magnus Carlsen, is rated at 2839. That gap implies Carlson would lose the overwhelming majority of games he played against Stockfish.

The defeat and subsequent humiliation of human chess players by computer programs made a big impression on people thinking about the future of artificial intelligence. Chess featured prominently in philosopher Nick Bostrom’s 2014 book, Superintelligence.

“One might have thought that great chess playing requires being able to learn abstract concepts, think cleverly about strategy, compose flexible plans, make a wide range of ingenious logical deductions, and maybe even model one’s opponent’s thinking,” Bostrom wrote. “Not so. It turned out to be possible to build a perfectly fine chess engine around a special-purpose algorithm.”

“It is tempting to speculate that other capabilities—such as general reasoning ability, or some key ability involved in programming—might likewise be achievable through some surprisingly simple algorithm,” Bostrom added.

Bostrom’s thesis was that humanity’s defeat at chess was just the beginning. He predicted that in the decades to come, computers would become better than humans at almost every cognitive task. And not just a little bit better. As with chess, Bostrom thought computer capabilities would soar past those of humans on a wide range of tasks from scientific discovery to military conquest. Bostrom warned that advanced AI could escape from the lab and literally take over the world.

Bostrom’s book was hugely influential. Elon Musk praised it shortly after its publication, describing AI as “potentially more dangerous than nukes.” In early 2015, Sam Altman described the book as “the best thing I’ve seen” on superhuman machine intelligence, which Altman considered “the greatest threat to the continued existence of humanity.” Later that year, Musk and Altman teamed up with several others to create OpenAI in hopes of steering AI technology in a more benign direction.

Bostrom acolytes scored a victory last month when the Biden Administration issued a sweeping executive order on artificial intelligence. Among other things, the order requires the creators of large AI models to report on the results of red-team safety testing, including their potential to evade human control through “deception or obfuscation.”

I did not find Bostrom’s arguments convincing back in 2015, and I still don’t. I think the emergence of superhuman chess software has led a lot of people astray. Chess is a game of perfect information and simple, deterministic rules. This means that it’s always possible to make chess software more powerful with better algorithms and more computing power.

But most important cognitive tasks—from translating a document to inventing a new technology to running an organization—don’t work like this. For these tasks, knowledge is at least as important as computing power. And because knowledge isn’t fungible, we’re not going to get a single super-AI that’s better than humans at everything. Instead, we’ll get a lot of different AIs with different strengths and weaknesses, none of which will be able to take over the world.

How computers surpassed humans at chess

Photo by Marco Verch (CC BY 2.0).

It was December 2001 and I was a computer science major at the University of Minnesota. I had enrolled in an AI class and my three-person team was presenting our final project: a Java program that played chess.

We hooked my laptop up to the projector and let our professor play against our software while we did our presentation. But it soon became clear that people were more interested in the game than our slides. People started shouting out suggestions for the professor’s next move. Eventually we stopped talking and stood watching as our program backed his king into a corner and checkmated him.

“You get an A,” the professor said with a chuckle as we headed back to our seats.

Our victorious chess engine was based on a fairly simple algorithm:

• Make a list of all possible four-move sequences.

• Score each resulting board position, where a higher score reflects a better position for white.

• Working backwards, find the strongest move at each board position—first three moves out, then two, then one—assuming that white will choose the move that yields the highest score, while black will do the opposite.

Today’s best chess engines are far more sophisticated than this. My team’s software scored positions using a simple piece-counting rule (queen=9 points, rook=5 points, etc). Today’s best chess engines use sophisticated neural networks to evaluate board positions.

Modern chess engines can also look much farther into the future. This is partly because increased computing power allows them to evaluate more positions. It’s also because sophisticated algorithms allow chess engines to focus on the most “likely” move sequences. As a result, modern chess engines can “look ahead” dozens of moves—something no human player can do.

The worry for singularists like Bostrom is that a lot of other cognitive tasks will turn out like chess: after computers reach parity with human beings, they’ll continue getting better and soon humans will be far behind. The rapid progress of large language models over the last five years has intensified those concerns.

Scaling language models

In 2020, OpenAI published a landmark paper showing a strong relationship between the size of a transformer-based language model (like GPT-3) and its performance. The researchers found that larger models—those with more trainable parameters—were better able to predict the next word in a document. For a given model size, “architectural details such as network width or depth have minimal effects within a wide range.”

Larger models require more computing power to train, so the paper helped to set off a computing arms race that continues to this day. As language models got bigger, they seemed to be gaining new human-like abilities.

But the “scaling laws” OpenAI articulated in 2020 came with an important caveat: larger models need more training data. This point was underscored by a widely read 2022 paper from Google’s DeepMind, which found that “for every doubling of model size the number of training tokens should also be doubled.”

The numbers are mind-boggling. DeepMind built a model called Chinchilla and trained it with 1.4 trillion tokens. That huge training set enabled Chinchilla to outperform OpenAI’s GPT-3 (which was trained for 300 billion tokens) despite the fact that GPT-3 had more than twice as many parameters (175 billion) as Chinchilla (70 billion).

In short, to make powerful language models, you need a lot of computing power and you also need a lot of data. Which raises an important question: could we run out of training data?

Last year a nonprofit called Epoch AI published a paper estimating that roughly 9 trillion words of high-quality data were available for training. If large language models continue growing at recent rates, they’ll bump up against this limit around 2026.

It’s not clear how binding this constraint will ultimately be. One recent paper found that it’s possible to reuse the same text up to four times with little loss of performance.  It’s also possible we’ll find ways to automate the creation of training data—though early efforts in this direction haven’t gone well.

But at a minimum, the need for training data complicates naive assumptions that Moore’s law will inevitably push LLMs to human capabilities and beyond.

Knowledge, not data

So far I’ve been talking about data because that’s what most empirical research has focused on. But what ultimately matters isn’t the number of tokens in a training set, it’s the amount of useful information—knowledge—contained in training data.

I think it’s helpful here to draw an analogy to the learning process of human beings. Imagine a smart college freshman taking an introductory physics course. She may have a lot of what psychologists call fluid intelligence—a general capacity to learn new information and solve novel problems. But she won’t initially be very good at solving physics problems—which psychologists call crystalized intelligence. Her physics acumen will develop over time as she does physics coursework.

If she goes on to grad school the nature of her learning will change. She’ll transition from mostly learning about the discoveries of other physicists to trying to make discoveries of her own. Her understanding of the laws of physics will continue to advance, but much more slowly than when she was speed-running through the discoveries of Newton, Maxwell, Einstein and Bohr.

You could say this slowdown would occur because she “ran out of training data,” but that’s not quite right. There would still be plenty of physics textbooks she hasn’t read, but they’d largely cover material she already knows.

An LLM in training is a bit like a student with a very low fluid intelligence. An LLM often needs to read thousands of documents on a particular topic to start understanding it. Data scientists compensate for the poor learning capacity of LLMs by using enormous training sets that repeat important facts and concepts over and over.

I expect that in the coming years we’ll develop language models with higher fluid intelligence—models that can glean more information from fewer documents, the way people do. If that happens, the apparent shortage of training data could prove illusory.

But I think it’s a mistake to assume that removing the data bottleneck in this way would clear the way for models to zoom past humans and achieve superintelligence. Think again about our hypothetical graduate student. Let’s say that she was able to reach the frontiers of physics knowledge after reading 20 textbooks. Could she have achieved a superhuman understanding of physics by reading 200 textbooks? Obviously not. Those extra 180 textbooks contain a lot of words, they don’t contain very much knowledge she doesn’t already have.

So too with AI systems. I suspect that on many tasks, their performance will start to plateau around human-level performance. Not because they “run out of data,” but because they reached the frontiers of human knowledge.

Thomas Edison’s insight

The simplicity and predictability of chess allow computers to “look ahead” and anticipate the likely consequences of any potential move. Most real-world problems are not like that.

For example, there’s a famous military saying that “no plan survives contact with the enemy.” The world is complex, and military planners are invariably working with incomplete and inaccurate information. When a battle begins, they inevitably discover that some of their assumptions were wrong and the battle plays out in ways they didn’t anticipate.

Thomas Edison had a saying that expresses a similar idea: “genius is one percent inspiration and 99 percent perspiration.” Edison experimented with 1,600 different materials to find a good material for the filament in his most famous invention, the electric light bulb.

“I never had an idea in my life,” Edison once said. “My so-called inventions already existed in the environment—I took them out. I’ve created nothing. Nobody does. There’s no such thing as an idea being brain-born; everything comes from the outside.”

A similar view is widely held in today’s Silicon Valley startup scene. In his influential 2011 book The Lean Startup, Eric Ries urged entrepreneurs to quickly build a “minimum viable product” so they could start getting feedback from real customers. It’s easy to think of products customers might want, but the only way to find out if they actually want it is to build it and try to sell it.

These are not interactions that can be “gamed out” digitally the way a chess engine analyzes possible chess moves. They require interacting with the real world: launching military campaigns, building and testing prototypes, putting products in the hands of real customers. This kind of trial and error is slow and usually requires a lot of help from other people.

Singularists tend to neglect the importance of trial and error. Bostrom, for example, writes that a superintelligent AI system might use nanotechnology to take over the world. Specifically, he envisions it producing “a detailed blueprint for how to bootstrap from existing technology (such as biotechnology and protein engineering) to the constructor capabilities needed for high-throughput atomically precise manufacturing that would allow inexpensive fabrication of a much wider range of nanomechanical structures.”

Next he envisions the AI covertly manufacturing a huge number of nano-robots that could spread across the globe. He predicts that “at a pre-set time, nanofactories producing nerve gas or target-seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe.”

It is not clear to me if this type of nanotechnology is even physically possible. But assuming it is, Bostrom seems naive about the amount of work it takes to go from concept to execution. Not only would this kind of invention require years of painstaking trial and error in the physical world, I expect it would require complex and expensive machinery like the astronomically expensive machines used to manufacture today’s most advanced silicon chips.

I asked Bostrom about this in an October interview. He acknowledged it was a challenge, but argued that a superintelligent AI might be able to come up with designs that don’t require much trial and error:

If you are trying to build the molecular structure, maybe one particular structure depends on various empirical parameters, you have to craft it just right and measure it precisely. If you're smarter, you'll see there's a huge design space, and pick another structure that is more robust to uncertainty, that you can be quite confident would work. It’s a little bit like if you design a lego set or something like that, it kind of clicks into place, you might not have to measure after you put the right lego block in place.

I don’t doubt that future AI systems could accelerate progress in nanotechnology by suggesting novel approaches. But in Edison’s estimation, a good idea only gets you about 1 percent of the way to a useful invention. The other 99 percent of the work—the “perspiration”—is going to require a lot of human help. And people aren’t going to want to help out unless they understand what they’re being asked to do and why.

Two claims about superintelligence

It’s helpful to distinguish between two different claims that fall under the broad heading of superintelligence:

• The weak superintelligence thesis says that due to Moore’s Law, AI systems will eventually achieve superhuman performance on most or all cognitive tasks.

• The strong superintelligence thesis says that a single AI system will master a wide range of tasks—and do it so rapidly that it is able to take over the world.

Last week I shared an early draft of this essay with my friend Sam Hammond, who believes that artificial general intelligence is closer than you might think. In his response, Sam pressed me to take the weak superintelligence thesis seriously, asking "what special sauce do humans have for adding value to scientific and technological progress that cannot be recreated by machine intelligence?"

On a fundamental level, I don’t think humans have any “special sauce.” I think it’s entirely possible that AI systems will eventually surpass humans at almost all cognitive tasks. But it matters how—and how quickly—this happens. 

If intelligence is mostly a matter of computing power, that points toward a winner-take-all competition. The smartest AI will be able to generate huge profits, which it can plow into buying more GPUs so it can become even more intelligent. In this “fast takeoff” scenario, high intelligence would enable a single powerful AI system to prevail in everything from technological innovation to military strategy.

But things look different if you assume that knowledge, rather than computing power, is the main bottleneck to increasing intelligence. Computing power is fungible; knowledge is not. Extensive knowledge about French literature, for example, won’t help an AI system design better rockets or computer chips.

So if knowledge is the essential input for intelligence, power is likely to remain widely distributed. Rocket companies know a lot about rocket design, so the best AI for designing rockets is likely to be made by (or in partnership with) rocket companies. Chip companies know a lot about chip design, so the best AI for designing computer chips is likely to be made by (or in partnership with) chip companies.

Importantly, the world’s militaries and defense contractors know a lot about manufacturing weapons—and are unlikely to share their knowledge with a rogue AI looking to build a robot army.

Of course some organizations will fail to adapt and may be displaced by new “AI native” organizations, just as companies like Kodak and Blockbuster were driven into bankruptcy by earlier waves of digital technology. But many other incumbents will adapt and thrive in the AI era.

The result won’t be a “singleton” that takes over the world, as predicted by the strong superintelligence thesis. Rather, we’ll get a pluralistic and competitive economy that’s not too different from the one we have now.

In 1945, the economist Freidrich Hayek wrote an essay called “The Use of Knowledge in Society.” It was a critique of socialists who called for economies to be centrally planned by national governments. 

These socialists implicitly thought of a nation’s economy as a giant chess board. They envisioned government officials surveying the board and moving economic resources around like chess pieces. They thought brainy government officials would be able to allocate these resources more efficiently than private individuals and firms.

But Hayek predicted that the socialists’ schemes would fail because the knowledge required to make efficient economic decisions was spread out in bits and pieces across the economy. He argued that a decentralized market economy is needed to enable the “man on the spot” to put his knowledge and experience to use.

I see singularists as making a similar mistake. They assume that all problems can be solved with the application of enough brainpower. But for many problems, having the right knowledge matters more. And a lot of economically significant knowledge is not contained in any public data set. It’s locked up in the brains and private databases of millions of individuals and organizations spread across the economy and around the world.

Thanks to Sean Trott, Sam Hammond, and Benjamin Riley for helpful comments on this essay. And thanks to Sean and Nick Bostrom for suggesting the distinction between fluid and crystalized intelligence.