I appreciate you writing this article! I've been wondering what your thoughts are on AI risk ever since you started the blog.
As some background, I first encountered you on Full Stack Economics, and when you announced this blog I subscribed here as well. Thus far I've found it very well-written and informative. In particular I loved your deep dives into self-driving technology, and found them very useful for forming my own opinions in that arena. You're one of my primary sources for news on contemporary AI developments, and I really appreciate the blog.
With that context, I want to say that I found this article to be very disappointing. It barely engages with the arguments in favor of AI risk, either handwaving them away without justification or omitting them entirely. Several sections even contain relatively simple mathematical errors that have nothing to do with AI in particular.
I'm writing up this comment because I believe AI to be by far the most impactful technology on the horizon, and it's vital that we can make good predictions on its impacts. If AI is indeed a threat to humanity, that would eclipse the importance of nearly every other issue humanity faces, and would justify strong measures to prevent it. And if AI is *not* such a threat, it has the potential to end poverty and war, saving millions of lives. In the latter case, we have a responsibility to develop it as quickly as possible. Figuring out which prediction is correct is *really important*.
To address things one at a time:
Chess:
You say that people have been mislead by chess, because chess follows simple deterministic rules and can therefore be solved by algorithms, which doesn't apply to the real world. This is a category error; there's no sharp delineation between those two domains. The real-world, just like chess, follows a set of relatively simple deterministic rules called "physics". Each "move" leads to a known outcome, which can be brute-force searched.
The difference, of course, is that the real-world game tree is vastly larger. The average move in a game of chess has about 35 options, compared to the number of particles in the observable universe is about 10^80. However this is less relevant than you might think, since chess's game tree is *already* large enough to be intractable to brute-force searches deeper than just a few moves as in your computer science class. Chess-playing algorithms succeeded by doing aggressive tree-pruning to get the search space down to a manageable size, along with heuristic arguments hardcoded in by human experience.
The piece valuation you used in your program is exactly such a fuzzy heuristic; nothing in the rules of chess assigns a value of "5" to a rook, and the actual usefullness of a rook varies wildly based on the exact position. Humans played thousands of games of chess, learned via trial and error and intuition how useful each piece was relative to each other piece, and then hardcoded that into their computers. A chess-playing algorithm like yours is *already* doing exactly the sort of knowledge-based heuristic approach that you claim computers aren't good at.
Early chess-playing pioneers like Deep Blue did rely on humans to explicitly program in those heuristics; they weren't doing the foundational reasoning themselves. But that changed in 2017 with AlphaZero, which learns chess entirely from scratch via neural network. It trained by playing chess against itself for only 9 hours and was then pitted agains the best human-coded chess-playing program, StockFish. AlphaZero won 25 games to 3.
The sort of pure algorithmic approach to games that you describe can only be used on very simple games like tic-tac-toe, and most of the things that computers have recently started doing much better than humans at use fuzzy heuristics learned by trial and error, just like humans do. AlphaStar, for example, is a neural network that can play Starcraft better than almost all humans. (Starcraft has a vastly larger game tree than chess, being more akin to the real world in the precision with which different actions can differ, and is also a hidden-information game where the players have to reason probabilistically about what the opponents have access to or may do.) OpenAI Five does the same with Dota 2. And outside of video games, DALL-E has far surpassed human artists in generality and visual beauty and fidelity. (It's still very poor at understanding an English description and converting that to a conceptually corresponding image, but that's a different skill.)
Your understanding of the real world also seems quite simplistic in certain domains. You say "The simplicity and predictability of chess allow computers to “look ahead” and anticipate the likely consequences of any potential move. Most real-world problems are not like that." and talk about military planning as an example of this; much of military strategy is doing exactly what you claim they don't do! The field of mathematical game theory was developed largely as a way to predict the actions of other nation-states in response to possible decisions, just like one does in chess. As you point out, real-world planning is a partial information game rather than a perfect-information game like chess, but that doesn't really have anything to do with the ability to plan ahead. Planning ahead in a hidden-information game looks very much the same as in chess, except that you ascribe probabilities to each of your opponent's moves and calculate the move you can take with the highest expected value.
There's a reason why game theory and wargaming both have "game" in their names; there's no sharp delineation between "game" and "geopolitics"; they're both complicated systems of rules, agents, incentives, and payoffs. Geopolitics is the same kind of thing as board games, just a more complicated instance.
Knowledge vs. computation:
If I understand your argument correctly, it's that general artificial intelligence will require more training data than humans currently have available to give it, and that much of the data we do have is redundant.
I think you actually understate part of this argument. The first important question is whether neural networks are capable of general intelligence *at all*. Our understanding of the human brain is extremely poor, and while neural networks are similar to them in many ways, they're also different in many ways. It's entirely possible that no amount of training data could ever get a neural network to human-level intelligence. (For more on this I'd highly recommend the debate between Scott Alexander and Gary Marcus: https://www.astralcodexten.com/p/somewhat-contra-marcus-on-ai-scaling)
But assuming that neural networks are capable in theory of general intelligence, it seems unjustified to point to limited training data as a relevant constraint.
* You point to a paper that estimates we'll run out of training data by 2026. This may be true, but what about the ~2.5 years before that happens? We've already seen dramatic improvement from GPT-2 to GPT-4, and if there is some point at which the amount of training data becomes "enough", you haven't provided any estimate of where exactly that point is, and it's entirely possible that it's above GPT-4 but below the total amount of data we have to throw at GPT-5.
* Humans are generating data at a frantic rate that's only increasing as the internet plays a larger and larger part of our lives. We may "run out" of unused training data in 2026, but that would only limit growth in training dataset size to the amount of data that humanity produces in a year, which is... a lot. Even if the amount of data needed for GAI is above the 2026 threshold, we'll still get there eventually, potentially only a few years later.
* You focus on human-created data, such as English passages. This is presumably because current leading AI models are language models, which is because that's what people want. AIs that can predict human language would be very useful to humanity, so that's where most of the funding goes. But when we're talking about *general* intelligence, capable of reasoning about the world from first principles and learning in much the same way that a human baby does, why would it need to be training on human language to start out with? There's nothing fundamentally special about humans, we're just a particularly complicated part of physics. The Large Hadron Collider produces more than 1 petabyte of data *per day*. The Event Horizon Telescope collected 5.5 petabytes of data in April of 2018. What happens when someone pipes all of that into a massive AI model? Nobody's done it yet because anything short of general intelligence will be unhelpful to the physics community, so the funding just isn't there. But if the rapid pace of increasing interest in AI continues, someone will do it eventually, and an AI capable of predicting physics is also capable of predicting human behavior as a side effect, since humans run on physics.
(Continued in a reply, I ran into the comment length limit.)
I don't know anything about nanotechnology, so the extent of my engagement with that subject will be to point out that prediction markets currently put a 70% chance on sci-fi-style rapidly self-replicating and world-altering nanotechnology being possible within the laws of physics. (https://manifold.markets/IsaacKing/is-it-physically-possible-to-design?r=SXNhYWNLaW5n)
The much more important point is this: If a superintelligent AI exists in the real world and wants humanity out of its way, *we have already lost*. Whether it uses nanotechnology or some other method is irrelevant; maybe it can't built nanobots and it has to do it the old fashioned way by spending a few years manipulating human pasties into positions of political power, so what? Talking about specific weapon technologies is completely missing the point that in almost any contest between a vastly smarter entity and a vastly dumber entity, the smarter one is going to win. The dumber entity going "I can't think of how the superintelligence would beat me, so it must be impossible" is just further evidence of that entity's dumbness. I have no idea how Magnus Carlson would beat me if we were to play chess, I could not predict his moves in advance or even begin to explain his strategy, but beat me he would.
In general, expecting that you've thought of all possible vectors for attack in a complex system is extreme overconfidence. About AlphaStar, a professional Starcraft player said: "AlphaStar is an intriguing and unorthodox player — one with the reflexes and speed of the best pros but strategies and a style that are entirely its own. The way AlphaStar was trained, with agents competing against each other in a league, has resulted in gameplay that’s unimaginably unusual; it really makes you question how much of StarCraft’s diverse possibilities pro players have really explored." Anyone who has worked in computer security can tell you about "security mindset"; the understanding that you're going up against adversaries who are equally or more intelligent than you, and that the slightest gap in any system *will* be exploited. Just being "kinda sure" is unacceptable in these sorts of environments.
This whole section sounds like your plan is "build the malevolent superintelligence and just trust that no matter how much it wants to kill us, it won't be able to figure out a way to do it", which is just... a really bad plan.
General epistemics:
Many of your arguments strike me as very odd, even ignoring the specifics. Several appear to be of the form "AI wouldn't post a risk for several years, therefore it isn't a risk at all". For example you point out that developing advanced nanotechnology would likely take several years, and present this as though it's supposed to be reassuring. Same for the growth of training datasets, as I addressed earlier. While I'll take a little solace in the fact that I probably won't be dying next year, I'd like to live a lot longer than that. Risks that are 5, 10, or 15 years out are still very much worth worrying about in my book!
If a CIA analyst discovered a plot by Russia to invade the US, do you think they'd present this as "Russia is intending to destroy the United States, but don't worry, it's going to take them several years to scale up their manufacturing capacity before they can execute their plan."? Or would they present it as "RUSSIA IS INTENDING TO DESTROY THE UNITED STATES AND WE ONLY HAVE A FEW YEARS TO PREPARE, THIS IS AN EMERGENCY WE NEED TO GET ON THIS RIGHT NOW!"?
Separately, most of these arguments also take the form of "eh, seems unlikely, so it's not a problem". This is an exceedingly strange approach to take when what's at stake is all of humanity. Even if we generously assume that these considerations bring the risk down to just 1%, a 1% chance of everyone dying is equivalent to about 80 million deaths in expectation. Asteroid impact avoidance, one of the few types of existential risk that the government spends significant funding on, uses a much stricter cutoff. The Jet Propulsion Laboratory's Sentry monitoring system, for example, tracks near-Earth objects down to a 0.00001% chance of impact any time within the next hundred years. Or take a look at this exercise that NASA ran on a hypothetical emergency situation where a large asteroid is discovered to have a 1% chance of impact in October of 2036: https://cneos.jpl.nasa.gov/pd/cs/pdc23/PDC23-ImpactRisk-Epoch1.pdf (And this is for an asteroid that would "only" destroy an area the size of a single US state!)
Heck, even when you just consider just a single individual, 1% is still super high. The average healthy young American has a much less than 1% chance of dying in 10 years; you only start getting risks that high doing skydiving and other crazy stuff. People avoid asbestos like the plague just because of a few percentage point increase in cancer risk 20+ years down the road. I don't understand why we'd treat a risk to every human alive as being less important than a risk to a single human.
Final thoughts:
My point here is not that AI has a >90% or even >50% chance of wiping out humanity; much lower numbers seem reasonable to me. But the arguments presented in this article are deeply flawed, and show no such thing.
I'd encourage you to engage with the AI safety community in more depth, as they've put years of work into this field and have much more sophisticated models of what facts and discoveries would indicate a higher or lower amount of risk. Here's a list of high quality introductions to the subject that present the arguments why AI might be dangerous: https://manifold.markets/Nikos/best-existing-short-form-introducti and here's a list of counterarguments arguing that AI is unlikely to be all that dangerous: https://www.reddit.com/r/singularity/comments/143qbk7/best_rebuttals_of_the_doomer_case_against_ai/ (Though notably almost all of those authors would put at least a 1% chance on AI causing human extinction. The lowest I'm aware of is this summary: https://arxiv.org/pdf/2306.02519.pdf, which places it at 0.4%; still alarmingly high compared to our threshold for worrying about asteroid impacts.)
It is a good question if feeding LHC data into a large neural network would force it to develop physical theory or learn to "feel" what would happen. Like a human "feels" when jumping or swimming without knowledge of underlying physics. But "AI capable of predicting physics is also capable of predicting human behavior as a side effect, since humans run on physics." is an overstretch. A cutting edge narrow AI can predict protein folding based on amino-acids sequence. Predicting even how a they interact to build a single cell is way beyond reach. Not to mention predicting human behavior based on particle physics.
You're confusing "is programmed with the laws of physics" with "is a general intelligence trained on the laws of physics". If you hardcode a physics simulation, then yes, it's intractable to simulate a human, or even to simulate just a single molecule. That's not what I'm talking about here. I'm talking about a neural network that is trained on physics and learns to predict the outcome of all sorts of different physics and chemistry experiments. That cannot be done with a simple brute force search, it requires intelligence.
Just like the difference between AlphaZero and a program where you program in the rules of Chess and tell it to brute force the answer. (It won't get more than a few moves in.)
"There's a reason why game theory and wargaming both have "game" in their names; there's no sharp delineation between "game" and "geopolitics"; they're both complicated systems of rules, agents, incentives, and payoffs. Geopolitics is the same kind of thing as board games, just a more complicated instance." - that is a pretty controversial thesis. Most people would say that there is clear distinction between games and real activities. That is probably why being great at playing games isn't considered a strong qualification when applying for a job... I see 2 fundamental differences: 1) rules of the real life are unknown. Nobody expected that Russia would get so much bogged down in Ukraine. How the war is going to look like, how much particular weapon systems work against the others - this needs to be checked empirically. 2) the number of actors is enormous and even if in their sheer number they may be approximated, it isn't like gas particles, social movements are so unpredictable. Sociologist try to grasp what is happening now or in the past but predicting future is so difficult. Also transferability of knowledge even between games is limited. Mastering shapes of go doesn't make you expert in chess even though these are similar class of games.
They're not super relevant to human matters though, because we're emergent phenomena many, many, many, layers of abstraction above the core rules. Knowing the rules can rule out a few things, like faster-than-light information transmission and perpetual motion machines, but the behavior of complex systems like humans and human society are far too complex for a fundamental approach to be tractable. Similarly, knowing the rules to Starcraft can rule out some things (I don't have an example, I don't actually play the game), but it doesn't really help you predict exactly where your opponent might send their units or how fast they can build up a complex base.
Everything else you describe is exactly what I was saying; the real world is more complicated than the games we play, as it has a much larger number of agents and environmental interactions, but nothing is fundamentally different.
I would be willing to bet you at 2:1 odds that the average player in the top percentile of Go players is also within the top 20% of Chess players after controlling for previous experience in Chess. (Possibly with a brief dip at the extreme beginner end, due to having to unlearn their Go-specific instincts.) A Go player will not know the Chess-specific specific strategies, but they'll already have mastered the sort of analytic thinking that's necessary for the sort of simple algorithmic games like both Go and Chess are.
I'm sympathetic to the overall argument, but if one person could reach the frontiers of (written) knowledge in every field at the same time, they could probably come up with a lot of novel ideas. Actual academic disciplines remain very siloed, useful human lifespans are pretty short if you consider it takes maybe 10 years to reach the frontier of a PhD-narrow field, and the incentives are very much against reaching that frontier in (superficially) unrelated disciplines.
James Burke makes that argument with his "connective" approach to science. We had a boom of science when we had the greats from different talents meet in a room and discuss solving a problem. From my own experience, what is public is behind what is private. I have seen a PHD student interviewed that did not know our company had already made her thesis irrelevant.
Agree. Because building AIs takes a lot of resources, they will typically be legally owned and controlled by organisations with access to thousands of smart people. Corporations and governments are forms of superintelligence. They are unlikely to allow their assets to go off piste and start creating nanobot armies. Occasionally, an AI will be badly managed (I’m guessing most likely when the owner is a corporation or country run by a single person) and get out from under its owner. But it will still have all the other AIs and their owners to deal with.
The bad behavior is not a bug, it is the reason for the model's existence. The way AI takes over the world is by chatbots playing to the fears of people on social media.
On a fun note, I can picture different countries putting different chatbots on social media platforms, then having them get into epic arguments where they try to brainwash each other. The world does not have enough popcorn for such a spectacle.
> they will typically be legally owned and controlled by organisations with access to thousands of smart people
Sounds like you're "assuming what you're trying to prove". The primary concern is that the companies will not be able to control the advanced AIs they design, so of course when you handwave that concern away, the future looks promising!
Exactly. Singularists tend to think that AGI would be deployed as kind of global autonomous administrator, which is trained "human values" and we hope would take care of us, humans like parents take care of children. However, I think it is much more likely it will be just another tool used by corporations and other powerful organizations.
Computers are very good at (metaphorical) perspiration. Even if they plateau around human intelligence, LLMs seem roughly 100s of times faster than humans, and tireless. A hypothetical AI that plateaued around human-level would still look like a super-genius, just through being able to put in so much work in a given amount of time. If it had any goal that money and power could help with, it could surely figure out a way to get some, paying or persuading people to be its hands as needed. Soon enough, some AI has millions of copies running, all dividing up tasks and working 24/7 toward a shared goal.
That seems high-risk even if you’re right about the data plateau limiting the potential of any single copy. Okay, lots of companies would be running controlled AIs specialized on their own data at the same time — that’s starting already — but nothing about that is incompatible with takeover scenarios.
What I mean by "perspiration" here is doing physical experiments or otherwise interacting with the physical world. Say you're trying to design a new rocket. Obviously, a team of a million AIs running at 100x human thought could come up with a rocket design much faster than a team of regular humans could. But somebody still needs to build the thing and test it, and there's limited room for AI to speed up those steps.
Physical testing isn't a requirement, it's just a handy shortcut for situations where you don't have enough computing power to simulate everything. We do not currently have the ability to simulate a rocket launch to anywhere near the requisite level of fidelity, but that doesn't mean it can't be done in principle.
You are never going to simulate a rocket launch down to the atomic level. To make it computationally tractable, you've gotta make some simplifying assumptions. How do you know if you've made he right simplifying assumptions such that your simulation gives you the same results as a physical test? The only way to do that is to run the physical test and compare.
With more computing power you can build better simulations that let you run more simulations and fewer physical tests. But the physical world is far too complicated to ever reach the point where the physical tests are totally unnecessary.
Especially because you have no way of being sure that the rocket you're simulating is identical to the rocket you're building. For example maybe a supplier gives you inaccurate specs for one of your parts, and as a result the simulated version of the part behaves differently than the real version in a way that causes the rocket to fail.
> How do you know if you've made he right simplifying assumptions such that your simulation gives you the same results as a physical test? The only way to do that is to run the physical test and compare.
Or just look at past data! Humans have already run huge numbers of simulations and recorded the details, and in many industries we're now able to one-shot designs from simulation alone. An AI can take all of that existing simulation data into account in its designs, there's no reason it would need to start from scratch.
> the physical world is far too complicated to ever reach the point where the physical tests are totally unnecessary.
This assertion seems unsupported, and contradicted by existing evidence. Just look at any engineering project; we don't built skyscrapers by trying a bunch of prototypes, seeing them collapse, and then trying again. We have detained models of how the world works in the conceptual vicinity of skyscrapers, and can simulate everything necessary to get it right on the first try. This is true even for totally new skyscraper designs that are not copies of any building we've ever tried building before.
If I understand correctly, seems you agree that data trades off against tests, and that more data allows for fewer tests, but you think there's a sharp cutoff between 1 and 0. Why would that be the case? I don't understand how it could be true that sufficient planning can get you down to only 1 test necessary before a working product, but untrue that you can go to 0; I don't see a reason why that additional step is impossible.
Also keep in mind that when we're talking about learning from tests, that means gathering data. I perform 10 tests and eventually have a specific collection of information that allows me to build a working item. Your claim seems to be that it would be impossible to acquire that set of data in any way other than performing physical tests, which seems highly unlikely to me. There are almost always multiple valid ways to arrive at any deduction.
Sourcing parts is definitely a relevant challenge, but that's just a special case of "how will an AI physically influence the world?" If we're assuming that the AI has some way to do that, then it can inspect the part for defects, manufacture them itself, etc. just like a human does before building the working version of whatever they've been iterating on.
And going back to the original point about this limiting a superintelligence's ability to do destructive things, a superintelligence does not need to perform 0 tests, it just needs to perform a small enough number of tests and do them quickly enough to be able to outmaneuver humans, which the parallelization that Chris M mentions is going to be very good for.
Basically, if your claim is just the affecting the macroscopic physical world is slower than the milliseconds involved in normal software calculations, then sure, that's obviously true. But if the claim is that it's impossible for an AI to affect the macroscopic physical world any faster than humans can, I don't think you've provided any particular evidence in favor of that.
"Or just look at past data! Humans have already run huge numbers of simulations and recorded the details, and in many industries we're now able to one-shot designs from simulation alone."
Yes!! My article said "knowledge is at least as important as computing power." You said (I think?), "no you don't need real-world knowledge because you can just use simulations instead." I said, "yeah, but how do you know if your simulation is accurate?" And I think you just said "well you can just look at the knowledge human beings have accumulated from past real-world testing."
Which... yes! That's the point! Computing power is of limited utility unless it's combined with real-world knowledge, and in the early years of an AGI most of the necessary knowledge (like data from past rocket tests) would be under the control of human beings who won't want to share.
I think maybe our disagreement here is a matter of degree rather than of kind? You agree that knowledge is needed for an AI to reach or exceed superhuman performance on most tasks, and you agree that obtaining this knowledge will usually be a slower process than AlphaZero teaching itself via self-play. But you think it'll still be doable, whereas I think it's going to be such a slog that we don't need to worry about ASI taking over the world. Does that seem fair?
Some form of knowledge is needed, yeah. Any intelligent agent needs to have an understanding of the current state of the world in order to be able to affect it. That information doesn't *need* to come from humans, but using human data to start out with will likely make things go a little faster.
AlphaZero took about 9 hours. It certainly seems plausible to me that it could take longer than that for an AGI to boopstrap itself to superintelligence, though it's also plausible that it could go faster, since it can do much more intelligent optimizations than the ones humans hardcoded into AlphaZero. If it turns out that it takes 100x times longer and doesn't happen for a full month, would you find that particularly reassuring?
Oh, and note that superintelligence is not required for taking over the world. A ~130 IQ human with no intelligence augmentation whatsoever would still be capable of doing it if they had the ability to create arbitrarily many digital copies of themselves and run them at 100x speed. (You may find it a fun exercise to think about how you'd do it yourself.)
I'd say pretty much all disagreements are a matter of degree. Figuring out the correct degree (or even the correct order of magnitude) is the hard part. :)
Any AI would face competition from other AIs directed by humans, many of which will try to persuade us to do something. It isn't a new threat, platform providers have to deal with filtering spam and scam. And you cannot persuade a stranger who doesn't want to listen to you no matter how strong your arguments are
What a nice refreshening portion of common sense, thank you for this article.
"The result won’t be a “singleton” that takes over the world, as predicted by the strong superintelligence thesis. Rather, we’ll get a pluralistic and competitive economy that’s not too different from the one we have now."
That seems very plausible, many people assume that this future superintelligence will appear in the world similar to the current one so that it can for example freely hack into computer systems or easily make money. But it is far more likely that there will be many other AIs at different levels of power and specialization. Security errors will be solved by then. Likewise Mustafa Suleyman's idea for the test of Artificial Capable Intelligence (https://www.technologyreview.com/2023/07/14/1076296/mustafa-suleyman-my-new-turing-test-would-see-if-ai-can-make-1-million) is somehow naive. There won't be opportunities available for making $1m from $100,000 investments by using only this cutting-edge AIs. They will have been already taken by businessmen who want to make money.
Yes, someone else made this point, but I am reading the Coming Wave and wanted to say that they trained their first Go AI using materials, but their second, more formidable AI, was trained by playing itself repetitively. The AIs probably will be able to train themselves.
Eh, yes and no. We actually do have "world simulator" in the form of various physics simulations, but those are too low-level to lead to any intelligent agent with the amount of computing power we have available at the moment.
But that's rather besides the point, because you don't need to simulate the entire world in order to design a general-purpose optimization algorithm; only a large enough subset of it. You can pit the AI against itself in things like negotiation (https://s3.amazonaws.com/end-to-end-negotiator/end-to-end-negotiator.pdf), playing diplomacy, starcraft, etc. and these games are open-ended enough that developing a capability to model the other player's minds is the optimal strategy. Once that capability exists, it can be applied to minds in the real world just as easily to players in a game.
Yes, training AI on various games looks promising. The point is, AI cannot learn about the real world from the games like it could master the worlds of Starcraft or Dota. It is similar for humans - children play a lot but most knowledge of the world they need to learn from books, other people, etc.
No, I'm saying the opposite. Cross-domain learning is possible, and in fact that's what current public-use neutral networks do to some extent. They train on a specific sample of data, then are let loose on a much larger set of possible inputs, and while there are some errors here and there, by and large they do pretty well. The more different the domains the more difficult it will be to generalize, but when it comes specifically to modeling other intelligences, there's not much of a difference between an agent's actions in a complicated video game and an agent's actions in the real world; both involve a brain doing the same sort of reasoning.
I agree that cross-domain learning and learning how to model other intelligences seems promising. It may be applied to the real world. But the original comment in this thread is "The AIs probably will be able to train themselves.". As I understood it - the idea is to train AGI *solely* on some kind of simulation so that it can achieve superhuman performance like AlphaZero. This motive seems similar to pre-LLMs ideas for superintelligence. To that I replied: we don't have a world simulator. We also need some knowledge in form of text written by humans.
I think you're misunderstanding what I'm saying. My claim is that with sufficiently advanced algorithms and enough computing power, they can in fact train by playing against themselves; a full world simulator is not necessary. You need a sufficiently complicated simulation, but it can still be much simpler than the real world. The AI can learn about itself/its opponent in that simulation and apply that learning effectively in the real world, since it's just "model brains" and "recognize patterns" either way.
Any Turing-complete system is in the important respects isomorphic to any other Turing-complete system. The real world is such a Turing-complete system, as are many game systems. An AI that learns to do optimally in a complicated game against an intelligent opponent can apply the exact same strategies to the real world.
If you’re just talking about LLMs, I mostly agree with this post.
If you meant to be talking about "all future AI", then I'll start listing some things that (I claim) some future AI will definitely be able to do, that you seem to be assuming to be forever beyond the reach of AI.
One thing is: remote-operating robots. If you give a human an existing remote-controllable robot, and a few hours' practice, they'll be able to do things with it that are way beyond any current AI. But the human brain is an algorithm too. If the human brain can figure out how to remote-control a robot with minimal practice, some future AI will be able to at least as well and quickly. There's a popular idea that robotics is a very hard unsolved problem, but the only constraint is today’s lousy algorithms. Remote-controllable robots are pretty cheap and easy to mass-manufacture. It's just that the demand is currently almost zero, because if you're going to pay a human salary regardless, you get a human body for zero marginal cost. As soon as AI exists that can remote-pilot a robot as well as a human brain can, supply & demand of remote-control robots would skyrocket. If there are millions, then billions, then trillions, of AI-controlled robots in the world, each of which can do all the kinds of things that humans can do (yes, including on-the-job training), it’s not an “economy that’s not too different from the one we have now”, right?
Another thing is: founding companies, and hiring people. Even if remote-control robots didn’t exist, we already have a world full of people carrying cameras and microphones and not making optimal use of their time. An micro-manager AI could walk a person through the process of doing whatever experiments are useful to the AI.
I also note that Joseph Stalin had merely one human brain and no particular physical prowess, but was able to amass extraordinary power. How did he do that? Whatever your answer is, why can't a future AI do those kinds of things too?
If you have 3 hours, here's Carl Shulman walking through, in great detail, what "AI takeover" might look like (without any nanotech): https://www.dwarkeshpatel.com/p/carl-shulman-2 :) And see also my blog post that I linked at the top.
Hi Steve! There's a lot there. Let me focus on the point about robots.
I don't think the bottleneck to human-level robots is just about having better software. I need to do more reporting on this, but my understanding is that the human body has several capabilities that robots lack:
* The human hand is extremely versatile. Our fingers have many degrees of freedom, we have a very good sense of touch, and we're able to perform intricate motions on delicate objects.
* We're bipedal with an excellent sense of balance. This allows us to climb stairs and ladders, operate on uneven terrain, etc.
* We're capable of self-repair. Our bodies respond to routine stress (exercise) by getting stronger. They heal automatically to minor injuries.
So it's true that with better AI it would be pretty easy to put a wheeled robot on a factory floor and have it replace some human workers. But I don't think we're anywhere close to having robots who can replace plumbers, lumberjacks, or nurses.
Also, current robots are not self-repairing, so if you have a million robots operating in the field that means you're going to have 100,000 new jobs for people building and repairing robots.
Anyway, I don't mean to say that AI isn't going to change the economy—certainly it's possible that there will be billions of robots in the future, and that would certainly be a very different world. But I think the transition to this world is going to take decades, not years. And what I specifically don't think is going to happen is a "fast takeoff" where a single AI gets a head start over everyone else, gets more and more powerful, and winds up controlling all the robots. Ownership and control over robots is more likely to be widely dispersed the way ownership and control of computers are today.
Thanks for your reply! Maybe this will be helpful clarification:
* If you had written: "There's this absolutely crazy future we're heading towards, but it involves other future yet-to-be-invented AI, not LLMs," then I'd agree, and indeed I have written similar things myself.
* If you had written: "There's this absolutely crazy future we're heading towards, but the path from here to there doesn't look like one AI doing recursive self-improvement", then I'd be mostly in agreement, albeit less confident than you, and we could have some interesting discussions around the edges.
* If you had written: "There's this absolutely crazy future we're heading towards, but definitely not before 2050", then we could have an interesting discussion about how you came to be so confident about that. (I would have said "It might or might not be before 2050, hard to say.".) Maybe we could talk about things like how fast AI is or isn't progressing, and what robot factory scale-ups would entail, etc. I would also say that, in AI, there's a weird tendency to treat the 2050s (for example) as if it's infinitely far away, whereas in almost any other domain, from climate change to city planning to life planning etc., the 2050s are appropriately treated as a real-life decade to think about and plan for.
But what you wrote is none of those things. You only mention "a pluralistic and competitive economy that’s not too different from the one we have now". If you're trying to imagine a future world where there are AI algorithms that can do everything a human brain can do, and such algorithms have been for decades — and if the thing you're imagining does not feel like absolutely wild sci-fi stuff — then I think you're not thinking about it carefully enough.
For example, in today's world, a median human 7-year-old can do lots of things, but he would not be trusted to make any important decision about business, economy, or governance. And not only would nobody hire that 7yo into their office or factory, they would probably pay to keep him out, because he would probably only mess stuff up.
…And I expect that what's true of that median 7yo today, will be true of human adults in this future. (Or something worse.)
I think someone in 1900 could list analogous advantages of horses versus cars and other machines. Horses can repair themselves, and horses can trod across narrow uneven paths, and horses have a much more flexible behavioral repertoire than cars etc.
But we know what happened. Cars can't drive through uneven narrow trails, so we figured out ways to do the things we want to do that don't require going through uneven narrow trails, like bulldozers and roads and helicopters.
Or think about it like this: suppose omnipotent aliens told us that the fingers and legs of every human would painlessly and irreversibly fall off on Jan. 1 2040. Would humanity collectively say "Oh well, that's the end of civilization, let's enjoy it while it lasts."? No, we would invent tools etc. and figure out ways to get things done that don't require fingers and legs. It's a solvable problem, in a society full of enterprising and brilliant people. By the same token, if future AI wants to do X that is not immediately possible with existing robotics, well, they'll invent better robotics, or they'll find a way to do what they want to do that doesn't require doing X, right?
As a layman, I'm not sure I grasp the exact challenge with insufficient training data.
I recently did a deep dive into the OpenAI paper about DALL-E 3. The gist of it is that the team figured out that current text-to-image models were so poor at prompt adherence because of poorly labeled images in their training set. So they built a bespoke captioner that was specifically trained to create rich, descriptive captions that covered every detail of the image. They then recaptioned all of the images in the training set using this AI captioner.
In their subsequent tests, they found that using 95% synthetic data from this captioner massively outperformed lower ratios.
So I wonder why this isn't possible with LLM training. Could an LLM not be trained specifically to produce new content for other LLMs to be trained on, taking into account all the needed nuances, etc.?
I guess you have partially answered this with your metaphor about reading 20 books vs. 200 books. LLMs can write the same content in many different ways, but they'll fundamentally be the same underlying ideas. I am also vaguely familiar with the term "model collapse" in reference to LLMs being trained on LLM-produced data.
But I haven't yet come across a clear explainer for why the training data limitations are important. I might not be the only one. Perhaps something for a future post of yours.
This is a great suggestion, and I will keep it in mind for future stories.
But yes as you say I think the fundamental issue is that in most domains synthetic data can only summarize knowledge that's already in the training set. Right now, models are pretty bad at "absorbing" knowledge in their training set, and synthetic data might help with that problem. But it can't generate genuinely new knowledge—only interactions with the external world can do that.
It does make sense that no matter how you phrase and rephrase the underlying content, you're ultimately not contributing anything new to the knowledge base.
What is less clear to me is where the exact problem lies with the lack of training data. Is it the lack of "new" knowledge or is it purely a lack of alternatively phrased writing. Because if it's the latter, then synthetic data should inutitively be helpful, as it can be adjusted to create lots of different voices and styles to train future LLMs on.
But, of course, if it's the former, "the entirety of written human knowledge" seems to be a hard ceiling. Definitely curious to hear a thorough deep dive into this with your in-depth understanding behind it.
Daniel, great point about the bespoke captioner. I can't tell if the idea of summarizing text could help. The language model is already deep and sophisticated, so it might already contain the information that a summary would add.
Tim, I too would like to learn more along this line.
Having previously lived in Silicon Valley for almost a decade, I know some people who feel deeply pessimistic about the future because of concerns about superintelligent AI. I'm in the camp of - it's good for some people to put in some precautionary work. But seeing people I know feeling deeply pessimistic about the future - whether because of global warming, rising inequality, or in this case, foom and doom, I feel so sad. The more thoughtful critiques the better.
I strongly agree that if there's a way we can convince people via good arguments to be less worried about future technological capabilities, that would be a very positive development. I'm not convinced that this article is such a thoughtful critique; its argument are largely unfounded, and if anything it makes me *more* worried that these are the arguments being put forth for the "don't worry about AI" proposition. (Since if better arguments existed, we'd expect to see those instead.)
Yes, I somewhat agree. I thought about editing my comment after posting it, but decided to leave it alone. Closer to what I meant is good-faith critique, rather than thoughtful. Most ‘anti doom’ arguments are low quality or made in bad faith. This is better. Even better is possible. I’d say more, but it’s time to get back to the turkey.
I'm not responding to the interview request because I'm the counterfactual - AI has NOT been able to do a single thing I asked it to. (Which included such complicated tasks as, "summarize these meeting notes." I wasn't asking it to create art.)
I most liked the section on Knowledge, and would go further. A thing that humans are good at is infusing decisions and actions with values. A computer can come up with a pro/con sheet, and can suggest the best action given some desired outcome (like winning the chess game). And to some extent, algorithms can be weighted with the values of the creators. But I think there are a lot of nuanced situations where "reasonable minds can disagree" and while AI can game out possible outcomes, it can't make a value-laden CHOICE. I guess the word I've been looking for is "wisdom." I think AI can be intelligent, it can probably be sort-of knowledgeable, but I'm not at all convinced it can be wise, or can meaningfully infuse decisions with values.
I don't understand your claim here; when a computer makes a good chess move, how is that differentness from valuing winning the game and making choices that further its values?
No offense to Timothy; I think this piece is one of the least thought-out articles on this blog. His others have been very solid summaries of the state of some sub-area of AI, but this one doesn't seem to have been well-researched and contains significant errors.
The main claim here - that I haven't seen much emphasis on elsewhere - is the idea that knowledge aquisition will represent a significant bottleneck for the development of intelligent machines. I'd be more inclined to spin this the other way. Handling lots of knowledge is an area where machines already excel. Filtering, sorting, summarizing, synthesizing are all handled pretty well by machines today. Yes, there's some data locked up in data silos, but this is a problem for humans as well as machines. An intelligent machine could likely do a good job of incentivising the silos to slack off their hold on their data. Silo'd data is an impediment, but it is probably not a show-stopping problem.
It's important not to confuse data and knowledge. Knowledge can be things like "the rocket will explode if you use this kind of valve with this kind of fuel at this high temperatures," or "it's a bad idea to Mike and Sarah on a team together because they had a big fight two years ago," or "this customer is a fan of the Red Sox and doesn't like Donald Trump." Some knowledge is represented in a database, and others might be recorded in product manuals or strategy memos or whatever. But a lot of knowledge isn't written down anywhere—it's just in the heads of people who do a certain kind of work.
A lot of knowledge is also tacit—even the person who has the knowledge may not be able to articulate it explicitly. Someone might say "that design doesn't seem right, but I can't quite put my finger on why." Yet if you ignore hunches like that from an experienced engineer, often the results are bad.
So the issue isn't just "data locked up in data silos." Even if an AI system hacked into every corporate network and stole all of its databases and formal documentation, there would still be a huge amount of valuable information that workers would know and the AI wouldn't. And of course hacking into every corporate network in the world is a non-trivial undertaking in its own right.
The argument that data is in worker's heads is an issue that slows down automation. A very common way to deal with it is just to discard that information and reinvent it. For example, a lot of bank tellers and checkout assistants lost their jobs to computer systems which lacked a lot of their knowledge, but could still do a similar job. Factory workers were the same story. Anyway, the issue of knowledge being difficult to extract will no-doubt slow things down and result in less future shock and more time for adaptation. However, unless we figure out a plan for a merger, we still face much the same fundamental issue of sharing the planet with a superior, engineered species that we must somehow learn to deal with.
It's a really well-written and thought provoking piece, and I find a lot of AI existential risk scenarios intuitively unpersuasive. Still, I'd quibble a little with a couple of things, or at least draw different conclusions from them.
The distinction between knowledge and data is really valuable, and I take the point that "economically significant knowledge is... locked up in the brains and private databases of millions of individuals and organizations". But what if the majority of those individuals and orgs decide to partner with the same AI platform? Andrew Whitby's point - that knowledge in one domain will combine usefully with knowledge from other domain - seems to push in that direction. Isn't knowledge the ultimate game of network effects? And if so, wouldn't we expect it to tend towards monopoly, all else being equal?
I also really like the point that you need constant real-world feedback if you're going to implement plans with real-world effects. But lots of people want their AI's to have real world effects and are going to build the tools to make that happen! That pretty much what automation is: if I want a faster and more efficient process for testing new materials, managing traffic, converting browsers into buyers, or blowing up hostile aeroplanes, I have a powerful incentive to automate, and that is going to mean building as tight a feedback loop as possible between intervention, AI and results from my intervention. The more I can get humans out of that loop, the faster and less error prone my business process becomes.
It's a big, complicated world, and I'm not going to claim it's impossible to build an AI with any particular set of capabilities. What I'm trying to draw attention to is that knowledge gives human beings a fair amount of leverage. A bunch of companies could decide to pool their knowledge to build an AI that's more powerful than what any of them could build individually, but I think this is a very different scenario from the "fast takeoff" scenario where an AI uses self-improvement to become the most powerful entity in the world practically overnight.
Another important point is that a lot of knowledge is tacit—even the person who has knowledge sometimes finds it difficult to summarize in a form that others will find useful. So I absolutely expect a lot of data sharing in some domains, but I think there will be other domains where incumbent organizations have deep expertise that can't easily be transferred to an AI entity. It's complicated.
Wonderful article! A not insignificant amount of valuable knowledge is also ‘process knowledge’ which is gained via experience in the physical world and interactions with other humans. This is hard to write down, let alone database for an AI to learn.
Really appreciated this article pointing out distinctions that get overlooked when discussing the different kinds of knowledge that AI and humans can acquire. Small note: "Garry Kasparov" is the correct spelling of the former world chess champion. :)
I appreciate you writing this article! I've been wondering what your thoughts are on AI risk ever since you started the blog.
As some background, I first encountered you on Full Stack Economics, and when you announced this blog I subscribed here as well. Thus far I've found it very well-written and informative. In particular I loved your deep dives into self-driving technology, and found them very useful for forming my own opinions in that arena. You're one of my primary sources for news on contemporary AI developments, and I really appreciate the blog.
With that context, I want to say that I found this article to be very disappointing. It barely engages with the arguments in favor of AI risk, either handwaving them away without justification or omitting them entirely. Several sections even contain relatively simple mathematical errors that have nothing to do with AI in particular.
I'm writing up this comment because I believe AI to be by far the most impactful technology on the horizon, and it's vital that we can make good predictions on its impacts. If AI is indeed a threat to humanity, that would eclipse the importance of nearly every other issue humanity faces, and would justify strong measures to prevent it. And if AI is *not* such a threat, it has the potential to end poverty and war, saving millions of lives. In the latter case, we have a responsibility to develop it as quickly as possible. Figuring out which prediction is correct is *really important*.
To address things one at a time:
Chess:
You say that people have been mislead by chess, because chess follows simple deterministic rules and can therefore be solved by algorithms, which doesn't apply to the real world. This is a category error; there's no sharp delineation between those two domains. The real-world, just like chess, follows a set of relatively simple deterministic rules called "physics". Each "move" leads to a known outcome, which can be brute-force searched.
The difference, of course, is that the real-world game tree is vastly larger. The average move in a game of chess has about 35 options, compared to the number of particles in the observable universe is about 10^80. However this is less relevant than you might think, since chess's game tree is *already* large enough to be intractable to brute-force searches deeper than just a few moves as in your computer science class. Chess-playing algorithms succeeded by doing aggressive tree-pruning to get the search space down to a manageable size, along with heuristic arguments hardcoded in by human experience.
The piece valuation you used in your program is exactly such a fuzzy heuristic; nothing in the rules of chess assigns a value of "5" to a rook, and the actual usefullness of a rook varies wildly based on the exact position. Humans played thousands of games of chess, learned via trial and error and intuition how useful each piece was relative to each other piece, and then hardcoded that into their computers. A chess-playing algorithm like yours is *already* doing exactly the sort of knowledge-based heuristic approach that you claim computers aren't good at.
Early chess-playing pioneers like Deep Blue did rely on humans to explicitly program in those heuristics; they weren't doing the foundational reasoning themselves. But that changed in 2017 with AlphaZero, which learns chess entirely from scratch via neural network. It trained by playing chess against itself for only 9 hours and was then pitted agains the best human-coded chess-playing program, StockFish. AlphaZero won 25 games to 3.
The sort of pure algorithmic approach to games that you describe can only be used on very simple games like tic-tac-toe, and most of the things that computers have recently started doing much better than humans at use fuzzy heuristics learned by trial and error, just like humans do. AlphaStar, for example, is a neural network that can play Starcraft better than almost all humans. (Starcraft has a vastly larger game tree than chess, being more akin to the real world in the precision with which different actions can differ, and is also a hidden-information game where the players have to reason probabilistically about what the opponents have access to or may do.) OpenAI Five does the same with Dota 2. And outside of video games, DALL-E has far surpassed human artists in generality and visual beauty and fidelity. (It's still very poor at understanding an English description and converting that to a conceptually corresponding image, but that's a different skill.)
Your understanding of the real world also seems quite simplistic in certain domains. You say "The simplicity and predictability of chess allow computers to “look ahead” and anticipate the likely consequences of any potential move. Most real-world problems are not like that." and talk about military planning as an example of this; much of military strategy is doing exactly what you claim they don't do! The field of mathematical game theory was developed largely as a way to predict the actions of other nation-states in response to possible decisions, just like one does in chess. As you point out, real-world planning is a partial information game rather than a perfect-information game like chess, but that doesn't really have anything to do with the ability to plan ahead. Planning ahead in a hidden-information game looks very much the same as in chess, except that you ascribe probabilities to each of your opponent's moves and calculate the move you can take with the highest expected value.
There's a reason why game theory and wargaming both have "game" in their names; there's no sharp delineation between "game" and "geopolitics"; they're both complicated systems of rules, agents, incentives, and payoffs. Geopolitics is the same kind of thing as board games, just a more complicated instance.
Knowledge vs. computation:
If I understand your argument correctly, it's that general artificial intelligence will require more training data than humans currently have available to give it, and that much of the data we do have is redundant.
I think you actually understate part of this argument. The first important question is whether neural networks are capable of general intelligence *at all*. Our understanding of the human brain is extremely poor, and while neural networks are similar to them in many ways, they're also different in many ways. It's entirely possible that no amount of training data could ever get a neural network to human-level intelligence. (For more on this I'd highly recommend the debate between Scott Alexander and Gary Marcus: https://www.astralcodexten.com/p/somewhat-contra-marcus-on-ai-scaling)
But assuming that neural networks are capable in theory of general intelligence, it seems unjustified to point to limited training data as a relevant constraint.
* You point to a paper that estimates we'll run out of training data by 2026. This may be true, but what about the ~2.5 years before that happens? We've already seen dramatic improvement from GPT-2 to GPT-4, and if there is some point at which the amount of training data becomes "enough", you haven't provided any estimate of where exactly that point is, and it's entirely possible that it's above GPT-4 but below the total amount of data we have to throw at GPT-5.
* Humans are generating data at a frantic rate that's only increasing as the internet plays a larger and larger part of our lives. We may "run out" of unused training data in 2026, but that would only limit growth in training dataset size to the amount of data that humanity produces in a year, which is... a lot. Even if the amount of data needed for GAI is above the 2026 threshold, we'll still get there eventually, potentially only a few years later.
* You focus on human-created data, such as English passages. This is presumably because current leading AI models are language models, which is because that's what people want. AIs that can predict human language would be very useful to humanity, so that's where most of the funding goes. But when we're talking about *general* intelligence, capable of reasoning about the world from first principles and learning in much the same way that a human baby does, why would it need to be training on human language to start out with? There's nothing fundamentally special about humans, we're just a particularly complicated part of physics. The Large Hadron Collider produces more than 1 petabyte of data *per day*. The Event Horizon Telescope collected 5.5 petabytes of data in April of 2018. What happens when someone pipes all of that into a massive AI model? Nobody's done it yet because anything short of general intelligence will be unhelpful to the physics community, so the funding just isn't there. But if the rapid pace of increasing interest in AI continues, someone will do it eventually, and an AI capable of predicting physics is also capable of predicting human behavior as a side effect, since humans run on physics.
(Continued in a reply, I ran into the comment length limit.)
Nanotechnology:
I don't know anything about nanotechnology, so the extent of my engagement with that subject will be to point out that prediction markets currently put a 70% chance on sci-fi-style rapidly self-replicating and world-altering nanotechnology being possible within the laws of physics. (https://manifold.markets/IsaacKing/is-it-physically-possible-to-design?r=SXNhYWNLaW5n)
The much more important point is this: If a superintelligent AI exists in the real world and wants humanity out of its way, *we have already lost*. Whether it uses nanotechnology or some other method is irrelevant; maybe it can't built nanobots and it has to do it the old fashioned way by spending a few years manipulating human pasties into positions of political power, so what? Talking about specific weapon technologies is completely missing the point that in almost any contest between a vastly smarter entity and a vastly dumber entity, the smarter one is going to win. The dumber entity going "I can't think of how the superintelligence would beat me, so it must be impossible" is just further evidence of that entity's dumbness. I have no idea how Magnus Carlson would beat me if we were to play chess, I could not predict his moves in advance or even begin to explain his strategy, but beat me he would.
In general, expecting that you've thought of all possible vectors for attack in a complex system is extreme overconfidence. About AlphaStar, a professional Starcraft player said: "AlphaStar is an intriguing and unorthodox player — one with the reflexes and speed of the best pros but strategies and a style that are entirely its own. The way AlphaStar was trained, with agents competing against each other in a league, has resulted in gameplay that’s unimaginably unusual; it really makes you question how much of StarCraft’s diverse possibilities pro players have really explored." Anyone who has worked in computer security can tell you about "security mindset"; the understanding that you're going up against adversaries who are equally or more intelligent than you, and that the slightest gap in any system *will* be exploited. Just being "kinda sure" is unacceptable in these sorts of environments.
This whole section sounds like your plan is "build the malevolent superintelligence and just trust that no matter how much it wants to kill us, it won't be able to figure out a way to do it", which is just... a really bad plan.
General epistemics:
Many of your arguments strike me as very odd, even ignoring the specifics. Several appear to be of the form "AI wouldn't post a risk for several years, therefore it isn't a risk at all". For example you point out that developing advanced nanotechnology would likely take several years, and present this as though it's supposed to be reassuring. Same for the growth of training datasets, as I addressed earlier. While I'll take a little solace in the fact that I probably won't be dying next year, I'd like to live a lot longer than that. Risks that are 5, 10, or 15 years out are still very much worth worrying about in my book!
If a CIA analyst discovered a plot by Russia to invade the US, do you think they'd present this as "Russia is intending to destroy the United States, but don't worry, it's going to take them several years to scale up their manufacturing capacity before they can execute their plan."? Or would they present it as "RUSSIA IS INTENDING TO DESTROY THE UNITED STATES AND WE ONLY HAVE A FEW YEARS TO PREPARE, THIS IS AN EMERGENCY WE NEED TO GET ON THIS RIGHT NOW!"?
Separately, most of these arguments also take the form of "eh, seems unlikely, so it's not a problem". This is an exceedingly strange approach to take when what's at stake is all of humanity. Even if we generously assume that these considerations bring the risk down to just 1%, a 1% chance of everyone dying is equivalent to about 80 million deaths in expectation. Asteroid impact avoidance, one of the few types of existential risk that the government spends significant funding on, uses a much stricter cutoff. The Jet Propulsion Laboratory's Sentry monitoring system, for example, tracks near-Earth objects down to a 0.00001% chance of impact any time within the next hundred years. Or take a look at this exercise that NASA ran on a hypothetical emergency situation where a large asteroid is discovered to have a 1% chance of impact in October of 2036: https://cneos.jpl.nasa.gov/pd/cs/pdc23/PDC23-ImpactRisk-Epoch1.pdf (And this is for an asteroid that would "only" destroy an area the size of a single US state!)
Heck, even when you just consider just a single individual, 1% is still super high. The average healthy young American has a much less than 1% chance of dying in 10 years; you only start getting risks that high doing skydiving and other crazy stuff. People avoid asbestos like the plague just because of a few percentage point increase in cancer risk 20+ years down the road. I don't understand why we'd treat a risk to every human alive as being less important than a risk to a single human.
Final thoughts:
My point here is not that AI has a >90% or even >50% chance of wiping out humanity; much lower numbers seem reasonable to me. But the arguments presented in this article are deeply flawed, and show no such thing.
I'd encourage you to engage with the AI safety community in more depth, as they've put years of work into this field and have much more sophisticated models of what facts and discoveries would indicate a higher or lower amount of risk. Here's a list of high quality introductions to the subject that present the arguments why AI might be dangerous: https://manifold.markets/Nikos/best-existing-short-form-introducti and here's a list of counterarguments arguing that AI is unlikely to be all that dangerous: https://www.reddit.com/r/singularity/comments/143qbk7/best_rebuttals_of_the_doomer_case_against_ai/ (Though notably almost all of those authors would put at least a 1% chance on AI causing human extinction. The lowest I'm aware of is this summary: https://arxiv.org/pdf/2306.02519.pdf, which places it at 0.4%; still alarmingly high compared to our threshold for worrying about asteroid impacts.)
Hi Issac! Thanks for the thoughtful and thorough comment. I'm not ignoring you, just want to give this the detailed response it deserves.
It is a good question if feeding LHC data into a large neural network would force it to develop physical theory or learn to "feel" what would happen. Like a human "feels" when jumping or swimming without knowledge of underlying physics. But "AI capable of predicting physics is also capable of predicting human behavior as a side effect, since humans run on physics." is an overstretch. A cutting edge narrow AI can predict protein folding based on amino-acids sequence. Predicting even how a they interact to build a single cell is way beyond reach. Not to mention predicting human behavior based on particle physics.
You're confusing "is programmed with the laws of physics" with "is a general intelligence trained on the laws of physics". If you hardcode a physics simulation, then yes, it's intractable to simulate a human, or even to simulate just a single molecule. That's not what I'm talking about here. I'm talking about a neural network that is trained on physics and learns to predict the outcome of all sorts of different physics and chemistry experiments. That cannot be done with a simple brute force search, it requires intelligence.
Just like the difference between AlphaZero and a program where you program in the rules of Chess and tell it to brute force the answer. (It won't get more than a few moves in.)
"There's a reason why game theory and wargaming both have "game" in their names; there's no sharp delineation between "game" and "geopolitics"; they're both complicated systems of rules, agents, incentives, and payoffs. Geopolitics is the same kind of thing as board games, just a more complicated instance." - that is a pretty controversial thesis. Most people would say that there is clear distinction between games and real activities. That is probably why being great at playing games isn't considered a strong qualification when applying for a job... I see 2 fundamental differences: 1) rules of the real life are unknown. Nobody expected that Russia would get so much bogged down in Ukraine. How the war is going to look like, how much particular weapon systems work against the others - this needs to be checked empirically. 2) the number of actors is enormous and even if in their sheer number they may be approximated, it isn't like gas particles, social movements are so unpredictable. Sociologist try to grasp what is happening now or in the past but predicting future is so difficult. Also transferability of knowledge even between games is limited. Mastering shapes of go doesn't make you expert in chess even though these are similar class of games.
On the contrary, the rules for real life are quite well known. They're available here: https://en.wikipedia.org/wiki/Quantum_mechanics
They're not super relevant to human matters though, because we're emergent phenomena many, many, many, layers of abstraction above the core rules. Knowing the rules can rule out a few things, like faster-than-light information transmission and perpetual motion machines, but the behavior of complex systems like humans and human society are far too complex for a fundamental approach to be tractable. Similarly, knowing the rules to Starcraft can rule out some things (I don't have an example, I don't actually play the game), but it doesn't really help you predict exactly where your opponent might send their units or how fast they can build up a complex base.
Everything else you describe is exactly what I was saying; the real world is more complicated than the games we play, as it has a much larger number of agents and environmental interactions, but nothing is fundamentally different.
I would be willing to bet you at 2:1 odds that the average player in the top percentile of Go players is also within the top 20% of Chess players after controlling for previous experience in Chess. (Possibly with a brief dip at the extreme beginner end, due to having to unlearn their Go-specific instincts.) A Go player will not know the Chess-specific specific strategies, but they'll already have mastered the sort of analytic thinking that's necessary for the sort of simple algorithmic games like both Go and Chess are.
I'm sympathetic to the overall argument, but if one person could reach the frontiers of (written) knowledge in every field at the same time, they could probably come up with a lot of novel ideas. Actual academic disciplines remain very siloed, useful human lifespans are pretty short if you consider it takes maybe 10 years to reach the frontier of a PhD-narrow field, and the incentives are very much against reaching that frontier in (superficially) unrelated disciplines.
I agree!
James Burke makes that argument with his "connective" approach to science. We had a boom of science when we had the greats from different talents meet in a room and discuss solving a problem. From my own experience, what is public is behind what is private. I have seen a PHD student interviewed that did not know our company had already made her thesis irrelevant.
they are rebooting James Burke show "Connections"
https://arstechnica.com/science/2023/11/fans-of-connections-rejoice-rebooted-classic-sci-doc-series-returns-with-original-host/
Good essay! Inexact reasoning of lossy analogies is causing quite a bit of confusion in the space.
I think inexact reasoning by lossy analogy is happening in this very article, leading many of its conclusions to be wrong.
https://www.understandingai.org/p/why-im-not-afraid-of-superintelligent/comment/43742458
Agree. Because building AIs takes a lot of resources, they will typically be legally owned and controlled by organisations with access to thousands of smart people. Corporations and governments are forms of superintelligence. They are unlikely to allow their assets to go off piste and start creating nanobot armies. Occasionally, an AI will be badly managed (I’m guessing most likely when the owner is a corporation or country run by a single person) and get out from under its owner. But it will still have all the other AIs and their owners to deal with.
The bad behavior is not a bug, it is the reason for the model's existence. The way AI takes over the world is by chatbots playing to the fears of people on social media.
On a fun note, I can picture different countries putting different chatbots on social media platforms, then having them get into epic arguments where they try to brainwash each other. The world does not have enough popcorn for such a spectacle.
Selling the fear is always the key.
> they will typically be legally owned and controlled by organisations with access to thousands of smart people
Sounds like you're "assuming what you're trying to prove". The primary concern is that the companies will not be able to control the advanced AIs they design, so of course when you handwave that concern away, the future looks promising!
https://www.astralcodexten.com/p/perhaps-it-is-a-bad-thing-that-the
Exactly. Singularists tend to think that AGI would be deployed as kind of global autonomous administrator, which is trained "human values" and we hope would take care of us, humans like parents take care of children. However, I think it is much more likely it will be just another tool used by corporations and other powerful organizations.
“Genius is 99 percent perspiration”
Computers are very good at (metaphorical) perspiration. Even if they plateau around human intelligence, LLMs seem roughly 100s of times faster than humans, and tireless. A hypothetical AI that plateaued around human-level would still look like a super-genius, just through being able to put in so much work in a given amount of time. If it had any goal that money and power could help with, it could surely figure out a way to get some, paying or persuading people to be its hands as needed. Soon enough, some AI has millions of copies running, all dividing up tasks and working 24/7 toward a shared goal.
That seems high-risk even if you’re right about the data plateau limiting the potential of any single copy. Okay, lots of companies would be running controlled AIs specialized on their own data at the same time — that’s starting already — but nothing about that is incompatible with takeover scenarios.
What I mean by "perspiration" here is doing physical experiments or otherwise interacting with the physical world. Say you're trying to design a new rocket. Obviously, a team of a million AIs running at 100x human thought could come up with a rocket design much faster than a team of regular humans could. But somebody still needs to build the thing and test it, and there's limited room for AI to speed up those steps.
True enough! It is a strong point against very-fast-takeoff nanotech-type scenarios.
Physical testing isn't a requirement, it's just a handy shortcut for situations where you don't have enough computing power to simulate everything. We do not currently have the ability to simulate a rocket launch to anywhere near the requisite level of fidelity, but that doesn't mean it can't be done in principle.
You are never going to simulate a rocket launch down to the atomic level. To make it computationally tractable, you've gotta make some simplifying assumptions. How do you know if you've made he right simplifying assumptions such that your simulation gives you the same results as a physical test? The only way to do that is to run the physical test and compare.
With more computing power you can build better simulations that let you run more simulations and fewer physical tests. But the physical world is far too complicated to ever reach the point where the physical tests are totally unnecessary.
Especially because you have no way of being sure that the rocket you're simulating is identical to the rocket you're building. For example maybe a supplier gives you inaccurate specs for one of your parts, and as a result the simulated version of the part behaves differently than the real version in a way that causes the rocket to fail.
> How do you know if you've made he right simplifying assumptions such that your simulation gives you the same results as a physical test? The only way to do that is to run the physical test and compare.
Or just look at past data! Humans have already run huge numbers of simulations and recorded the details, and in many industries we're now able to one-shot designs from simulation alone. An AI can take all of that existing simulation data into account in its designs, there's no reason it would need to start from scratch.
> the physical world is far too complicated to ever reach the point where the physical tests are totally unnecessary.
This assertion seems unsupported, and contradicted by existing evidence. Just look at any engineering project; we don't built skyscrapers by trying a bunch of prototypes, seeing them collapse, and then trying again. We have detained models of how the world works in the conceptual vicinity of skyscrapers, and can simulate everything necessary to get it right on the first try. This is true even for totally new skyscraper designs that are not copies of any building we've ever tried building before.
If I understand correctly, seems you agree that data trades off against tests, and that more data allows for fewer tests, but you think there's a sharp cutoff between 1 and 0. Why would that be the case? I don't understand how it could be true that sufficient planning can get you down to only 1 test necessary before a working product, but untrue that you can go to 0; I don't see a reason why that additional step is impossible.
Also keep in mind that when we're talking about learning from tests, that means gathering data. I perform 10 tests and eventually have a specific collection of information that allows me to build a working item. Your claim seems to be that it would be impossible to acquire that set of data in any way other than performing physical tests, which seems highly unlikely to me. There are almost always multiple valid ways to arrive at any deduction.
Sourcing parts is definitely a relevant challenge, but that's just a special case of "how will an AI physically influence the world?" If we're assuming that the AI has some way to do that, then it can inspect the part for defects, manufacture them itself, etc. just like a human does before building the working version of whatever they've been iterating on.
And going back to the original point about this limiting a superintelligence's ability to do destructive things, a superintelligence does not need to perform 0 tests, it just needs to perform a small enough number of tests and do them quickly enough to be able to outmaneuver humans, which the parallelization that Chris M mentions is going to be very good for.
Basically, if your claim is just the affecting the macroscopic physical world is slower than the milliseconds involved in normal software calculations, then sure, that's obviously true. But if the claim is that it's impossible for an AI to affect the macroscopic physical world any faster than humans can, I don't think you've provided any particular evidence in favor of that.
"Or just look at past data! Humans have already run huge numbers of simulations and recorded the details, and in many industries we're now able to one-shot designs from simulation alone."
Yes!! My article said "knowledge is at least as important as computing power." You said (I think?), "no you don't need real-world knowledge because you can just use simulations instead." I said, "yeah, but how do you know if your simulation is accurate?" And I think you just said "well you can just look at the knowledge human beings have accumulated from past real-world testing."
Which... yes! That's the point! Computing power is of limited utility unless it's combined with real-world knowledge, and in the early years of an AGI most of the necessary knowledge (like data from past rocket tests) would be under the control of human beings who won't want to share.
I think maybe our disagreement here is a matter of degree rather than of kind? You agree that knowledge is needed for an AI to reach or exceed superhuman performance on most tasks, and you agree that obtaining this knowledge will usually be a slower process than AlphaZero teaching itself via self-play. But you think it'll still be doable, whereas I think it's going to be such a slog that we don't need to worry about ASI taking over the world. Does that seem fair?
Some form of knowledge is needed, yeah. Any intelligent agent needs to have an understanding of the current state of the world in order to be able to affect it. That information doesn't *need* to come from humans, but using human data to start out with will likely make things go a little faster.
AlphaZero took about 9 hours. It certainly seems plausible to me that it could take longer than that for an AGI to boopstrap itself to superintelligence, though it's also plausible that it could go faster, since it can do much more intelligent optimizations than the ones humans hardcoded into AlphaZero. If it turns out that it takes 100x times longer and doesn't happen for a full month, would you find that particularly reassuring?
Oh, and note that superintelligence is not required for taking over the world. A ~130 IQ human with no intelligence augmentation whatsoever would still be capable of doing it if they had the ability to create arbitrarily many digital copies of themselves and run them at 100x speed. (You may find it a fun exercise to think about how you'd do it yourself.)
I'd say pretty much all disagreements are a matter of degree. Figuring out the correct degree (or even the correct order of magnitude) is the hard part. :)
Any AI would face competition from other AIs directed by humans, many of which will try to persuade us to do something. It isn't a new threat, platform providers have to deal with filtering spam and scam. And you cannot persuade a stranger who doesn't want to listen to you no matter how strong your arguments are
Why do you think humans wouldn't want to listen to AI?
What a nice refreshening portion of common sense, thank you for this article.
"The result won’t be a “singleton” that takes over the world, as predicted by the strong superintelligence thesis. Rather, we’ll get a pluralistic and competitive economy that’s not too different from the one we have now."
That seems very plausible, many people assume that this future superintelligence will appear in the world similar to the current one so that it can for example freely hack into computer systems or easily make money. But it is far more likely that there will be many other AIs at different levels of power and specialization. Security errors will be solved by then. Likewise Mustafa Suleyman's idea for the test of Artificial Capable Intelligence (https://www.technologyreview.com/2023/07/14/1076296/mustafa-suleyman-my-new-turing-test-would-see-if-ai-can-make-1-million) is somehow naive. There won't be opportunities available for making $1m from $100,000 investments by using only this cutting-edge AIs. They will have been already taken by businessmen who want to make money.
Yes, someone else made this point, but I am reading the Coming Wave and wanted to say that they trained their first Go AI using materials, but their second, more formidable AI, was trained by playing itself repetitively. The AIs probably will be able to train themselves.
Crucial difference is that we don't have anything resembling World Simulator just like we had a virtual go board.
Yes this is a really important point. I think I'm going to do a full post about it actually.
Eh, yes and no. We actually do have "world simulator" in the form of various physics simulations, but those are too low-level to lead to any intelligent agent with the amount of computing power we have available at the moment.
But that's rather besides the point, because you don't need to simulate the entire world in order to design a general-purpose optimization algorithm; only a large enough subset of it. You can pit the AI against itself in things like negotiation (https://s3.amazonaws.com/end-to-end-negotiator/end-to-end-negotiator.pdf), playing diplomacy, starcraft, etc. and these games are open-ended enough that developing a capability to model the other player's minds is the optimal strategy. Once that capability exists, it can be applied to minds in the real world just as easily to players in a game.
Yes, training AI on various games looks promising. The point is, AI cannot learn about the real world from the games like it could master the worlds of Starcraft or Dota. It is similar for humans - children play a lot but most knowledge of the world they need to learn from books, other people, etc.
No, I'm saying the opposite. Cross-domain learning is possible, and in fact that's what current public-use neutral networks do to some extent. They train on a specific sample of data, then are let loose on a much larger set of possible inputs, and while there are some errors here and there, by and large they do pretty well. The more different the domains the more difficult it will be to generalize, but when it comes specifically to modeling other intelligences, there's not much of a difference between an agent's actions in a complicated video game and an agent's actions in the real world; both involve a brain doing the same sort of reasoning.
I agree that cross-domain learning and learning how to model other intelligences seems promising. It may be applied to the real world. But the original comment in this thread is "The AIs probably will be able to train themselves.". As I understood it - the idea is to train AGI *solely* on some kind of simulation so that it can achieve superhuman performance like AlphaZero. This motive seems similar to pre-LLMs ideas for superintelligence. To that I replied: we don't have a world simulator. We also need some knowledge in form of text written by humans.
I think you're misunderstanding what I'm saying. My claim is that with sufficiently advanced algorithms and enough computing power, they can in fact train by playing against themselves; a full world simulator is not necessary. You need a sufficiently complicated simulation, but it can still be much simpler than the real world. The AI can learn about itself/its opponent in that simulation and apply that learning effectively in the real world, since it's just "model brains" and "recognize patterns" either way.
Any Turing-complete system is in the important respects isomorphic to any other Turing-complete system. The real world is such a Turing-complete system, as are many game systems. An AI that learns to do optimally in a complicated game against an intelligent opponent can apply the exact same strategies to the real world.
You seem to be talking about "all future AI", but I get a vibe that you're really thinking about LLMs. See the table at the top of this blog post: https://www.alignmentforum.org/posts/rgPxEKFBLpLqJpMBM/response-to-blake-richards-agi-generality-alignment-and-loss
If you’re just talking about LLMs, I mostly agree with this post.
If you meant to be talking about "all future AI", then I'll start listing some things that (I claim) some future AI will definitely be able to do, that you seem to be assuming to be forever beyond the reach of AI.
One thing is: remote-operating robots. If you give a human an existing remote-controllable robot, and a few hours' practice, they'll be able to do things with it that are way beyond any current AI. But the human brain is an algorithm too. If the human brain can figure out how to remote-control a robot with minimal practice, some future AI will be able to at least as well and quickly. There's a popular idea that robotics is a very hard unsolved problem, but the only constraint is today’s lousy algorithms. Remote-controllable robots are pretty cheap and easy to mass-manufacture. It's just that the demand is currently almost zero, because if you're going to pay a human salary regardless, you get a human body for zero marginal cost. As soon as AI exists that can remote-pilot a robot as well as a human brain can, supply & demand of remote-control robots would skyrocket. If there are millions, then billions, then trillions, of AI-controlled robots in the world, each of which can do all the kinds of things that humans can do (yes, including on-the-job training), it’s not an “economy that’s not too different from the one we have now”, right?
Another thing is: founding companies, and hiring people. Even if remote-control robots didn’t exist, we already have a world full of people carrying cameras and microphones and not making optimal use of their time. An micro-manager AI could walk a person through the process of doing whatever experiments are useful to the AI.
I also note that Joseph Stalin had merely one human brain and no particular physical prowess, but was able to amass extraordinary power. How did he do that? Whatever your answer is, why can't a future AI do those kinds of things too?
If you have 3 hours, here's Carl Shulman walking through, in great detail, what "AI takeover" might look like (without any nanotech): https://www.dwarkeshpatel.com/p/carl-shulman-2 :) And see also my blog post that I linked at the top.
Hi Steve! There's a lot there. Let me focus on the point about robots.
I don't think the bottleneck to human-level robots is just about having better software. I need to do more reporting on this, but my understanding is that the human body has several capabilities that robots lack:
* The human hand is extremely versatile. Our fingers have many degrees of freedom, we have a very good sense of touch, and we're able to perform intricate motions on delicate objects.
* We're bipedal with an excellent sense of balance. This allows us to climb stairs and ladders, operate on uneven terrain, etc.
* We're capable of self-repair. Our bodies respond to routine stress (exercise) by getting stronger. They heal automatically to minor injuries.
So it's true that with better AI it would be pretty easy to put a wheeled robot on a factory floor and have it replace some human workers. But I don't think we're anywhere close to having robots who can replace plumbers, lumberjacks, or nurses.
Also, current robots are not self-repairing, so if you have a million robots operating in the field that means you're going to have 100,000 new jobs for people building and repairing robots.
Anyway, I don't mean to say that AI isn't going to change the economy—certainly it's possible that there will be billions of robots in the future, and that would certainly be a very different world. But I think the transition to this world is going to take decades, not years. And what I specifically don't think is going to happen is a "fast takeoff" where a single AI gets a head start over everyone else, gets more and more powerful, and winds up controlling all the robots. Ownership and control over robots is more likely to be widely dispersed the way ownership and control of computers are today.
Thanks for your reply! Maybe this will be helpful clarification:
* If you had written: "There's this absolutely crazy future we're heading towards, but it involves other future yet-to-be-invented AI, not LLMs," then I'd agree, and indeed I have written similar things myself.
* If you had written: "There's this absolutely crazy future we're heading towards, but the path from here to there doesn't look like one AI doing recursive self-improvement", then I'd be mostly in agreement, albeit less confident than you, and we could have some interesting discussions around the edges.
* If you had written: "There's this absolutely crazy future we're heading towards, but definitely not before 2050", then we could have an interesting discussion about how you came to be so confident about that. (I would have said "It might or might not be before 2050, hard to say.".) Maybe we could talk about things like how fast AI is or isn't progressing, and what robot factory scale-ups would entail, etc. I would also say that, in AI, there's a weird tendency to treat the 2050s (for example) as if it's infinitely far away, whereas in almost any other domain, from climate change to city planning to life planning etc., the 2050s are appropriately treated as a real-life decade to think about and plan for.
But what you wrote is none of those things. You only mention "a pluralistic and competitive economy that’s not too different from the one we have now". If you're trying to imagine a future world where there are AI algorithms that can do everything a human brain can do, and such algorithms have been for decades — and if the thing you're imagining does not feel like absolutely wild sci-fi stuff — then I think you're not thinking about it carefully enough.
For example, in today's world, a median human 7-year-old can do lots of things, but he would not be trusted to make any important decision about business, economy, or governance. And not only would nobody hire that 7yo into their office or factory, they would probably pay to keep him out, because he would probably only mess stuff up.
…And I expect that what's true of that median 7yo today, will be true of human adults in this future. (Or something worse.)
See for example, Holden Karnofsky's "Most Important Century" series, e.g. https://www.cold-takes.com/how-digital-people-could-change-the-world/
As to your comments about robots:
I think someone in 1900 could list analogous advantages of horses versus cars and other machines. Horses can repair themselves, and horses can trod across narrow uneven paths, and horses have a much more flexible behavioral repertoire than cars etc.
But we know what happened. Cars can't drive through uneven narrow trails, so we figured out ways to do the things we want to do that don't require going through uneven narrow trails, like bulldozers and roads and helicopters.
I think there are already dexterous robot hands, e.g. https://www.shadowrobot.com/dexterous-hand-series/ which I heard about from https://openai.com/research/solving-rubiks-cube . Again, these are currently expensive niche products because there is almost no demand for them, because there is no AI that can use them well.
Or think about it like this: suppose omnipotent aliens told us that the fingers and legs of every human would painlessly and irreversibly fall off on Jan. 1 2040. Would humanity collectively say "Oh well, that's the end of civilization, let's enjoy it while it lasts."? No, we would invent tools etc. and figure out ways to get things done that don't require fingers and legs. It's a solvable problem, in a society full of enterprising and brilliant people. By the same token, if future AI wants to do X that is not immediately possible with existing robotics, well, they'll invent better robotics, or they'll find a way to do what they want to do that doesn't require doing X, right?
Hope this is helpful, happy to keep chatting :)
This was a fascinating read, thanks!
As a layman, I'm not sure I grasp the exact challenge with insufficient training data.
I recently did a deep dive into the OpenAI paper about DALL-E 3. The gist of it is that the team figured out that current text-to-image models were so poor at prompt adherence because of poorly labeled images in their training set. So they built a bespoke captioner that was specifically trained to create rich, descriptive captions that covered every detail of the image. They then recaptioned all of the images in the training set using this AI captioner.
In their subsequent tests, they found that using 95% synthetic data from this captioner massively outperformed lower ratios.
So I wonder why this isn't possible with LLM training. Could an LLM not be trained specifically to produce new content for other LLMs to be trained on, taking into account all the needed nuances, etc.?
I guess you have partially answered this with your metaphor about reading 20 books vs. 200 books. LLMs can write the same content in many different ways, but they'll fundamentally be the same underlying ideas. I am also vaguely familiar with the term "model collapse" in reference to LLMs being trained on LLM-produced data.
But I haven't yet come across a clear explainer for why the training data limitations are important. I might not be the only one. Perhaps something for a future post of yours.
This is a great suggestion, and I will keep it in mind for future stories.
But yes as you say I think the fundamental issue is that in most domains synthetic data can only summarize knowledge that's already in the training set. Right now, models are pretty bad at "absorbing" knowledge in their training set, and synthetic data might help with that problem. But it can't generate genuinely new knowledge—only interactions with the external world can do that.
Thank you for the response!
It does make sense that no matter how you phrase and rephrase the underlying content, you're ultimately not contributing anything new to the knowledge base.
What is less clear to me is where the exact problem lies with the lack of training data. Is it the lack of "new" knowledge or is it purely a lack of alternatively phrased writing. Because if it's the latter, then synthetic data should inutitively be helpful, as it can be adjusted to create lots of different voices and styles to train future LLMs on.
But, of course, if it's the former, "the entirety of written human knowledge" seems to be a hard ceiling. Definitely curious to hear a thorough deep dive into this with your in-depth understanding behind it.
Daniel, great point about the bespoke captioner. I can't tell if the idea of summarizing text could help. The language model is already deep and sophisticated, so it might already contain the information that a summary would add.
Tim, I too would like to learn more along this line.
Having previously lived in Silicon Valley for almost a decade, I know some people who feel deeply pessimistic about the future because of concerns about superintelligent AI. I'm in the camp of - it's good for some people to put in some precautionary work. But seeing people I know feeling deeply pessimistic about the future - whether because of global warming, rising inequality, or in this case, foom and doom, I feel so sad. The more thoughtful critiques the better.
I strongly agree that if there's a way we can convince people via good arguments to be less worried about future technological capabilities, that would be a very positive development. I'm not convinced that this article is such a thoughtful critique; its argument are largely unfounded, and if anything it makes me *more* worried that these are the arguments being put forth for the "don't worry about AI" proposition. (Since if better arguments existed, we'd expect to see those instead.)
https://www.understandingai.org/p/why-im-not-afraid-of-superintelligent/comment/43742458
Yes, I somewhat agree. I thought about editing my comment after posting it, but decided to leave it alone. Closer to what I meant is good-faith critique, rather than thoughtful. Most ‘anti doom’ arguments are low quality or made in bad faith. This is better. Even better is possible. I’d say more, but it’s time to get back to the turkey.
I'm not responding to the interview request because I'm the counterfactual - AI has NOT been able to do a single thing I asked it to. (Which included such complicated tasks as, "summarize these meeting notes." I wasn't asking it to create art.)
I most liked the section on Knowledge, and would go further. A thing that humans are good at is infusing decisions and actions with values. A computer can come up with a pro/con sheet, and can suggest the best action given some desired outcome (like winning the chess game). And to some extent, algorithms can be weighted with the values of the creators. But I think there are a lot of nuanced situations where "reasonable minds can disagree" and while AI can game out possible outcomes, it can't make a value-laden CHOICE. I guess the word I've been looking for is "wisdom." I think AI can be intelligent, it can probably be sort-of knowledgeable, but I'm not at all convinced it can be wise, or can meaningfully infuse decisions with values.
I don't understand your claim here; when a computer makes a good chess move, how is that differentness from valuing winning the game and making choices that further its values?
well thought out piece. appreciate the perspective and thoughtfulness
No offense to Timothy; I think this piece is one of the least thought-out articles on this blog. His others have been very solid summaries of the state of some sub-area of AI, but this one doesn't seem to have been well-researched and contains significant errors.
https://www.understandingai.org/p/why-im-not-afraid-of-superintelligent/comment/43742458
The main claim here - that I haven't seen much emphasis on elsewhere - is the idea that knowledge aquisition will represent a significant bottleneck for the development of intelligent machines. I'd be more inclined to spin this the other way. Handling lots of knowledge is an area where machines already excel. Filtering, sorting, summarizing, synthesizing are all handled pretty well by machines today. Yes, there's some data locked up in data silos, but this is a problem for humans as well as machines. An intelligent machine could likely do a good job of incentivising the silos to slack off their hold on their data. Silo'd data is an impediment, but it is probably not a show-stopping problem.
It's important not to confuse data and knowledge. Knowledge can be things like "the rocket will explode if you use this kind of valve with this kind of fuel at this high temperatures," or "it's a bad idea to Mike and Sarah on a team together because they had a big fight two years ago," or "this customer is a fan of the Red Sox and doesn't like Donald Trump." Some knowledge is represented in a database, and others might be recorded in product manuals or strategy memos or whatever. But a lot of knowledge isn't written down anywhere—it's just in the heads of people who do a certain kind of work.
A lot of knowledge is also tacit—even the person who has the knowledge may not be able to articulate it explicitly. Someone might say "that design doesn't seem right, but I can't quite put my finger on why." Yet if you ignore hunches like that from an experienced engineer, often the results are bad.
So the issue isn't just "data locked up in data silos." Even if an AI system hacked into every corporate network and stole all of its databases and formal documentation, there would still be a huge amount of valuable information that workers would know and the AI wouldn't. And of course hacking into every corporate network in the world is a non-trivial undertaking in its own right.
The argument that data is in worker's heads is an issue that slows down automation. A very common way to deal with it is just to discard that information and reinvent it. For example, a lot of bank tellers and checkout assistants lost their jobs to computer systems which lacked a lot of their knowledge, but could still do a similar job. Factory workers were the same story. Anyway, the issue of knowledge being difficult to extract will no-doubt slow things down and result in less future shock and more time for adaptation. However, unless we figure out a plan for a merger, we still face much the same fundamental issue of sharing the planet with a superior, engineered species that we must somehow learn to deal with.
It's a really well-written and thought provoking piece, and I find a lot of AI existential risk scenarios intuitively unpersuasive. Still, I'd quibble a little with a couple of things, or at least draw different conclusions from them.
The distinction between knowledge and data is really valuable, and I take the point that "economically significant knowledge is... locked up in the brains and private databases of millions of individuals and organizations". But what if the majority of those individuals and orgs decide to partner with the same AI platform? Andrew Whitby's point - that knowledge in one domain will combine usefully with knowledge from other domain - seems to push in that direction. Isn't knowledge the ultimate game of network effects? And if so, wouldn't we expect it to tend towards monopoly, all else being equal?
I also really like the point that you need constant real-world feedback if you're going to implement plans with real-world effects. But lots of people want their AI's to have real world effects and are going to build the tools to make that happen! That pretty much what automation is: if I want a faster and more efficient process for testing new materials, managing traffic, converting browsers into buyers, or blowing up hostile aeroplanes, I have a powerful incentive to automate, and that is going to mean building as tight a feedback loop as possible between intervention, AI and results from my intervention. The more I can get humans out of that loop, the faster and less error prone my business process becomes.
It's a big, complicated world, and I'm not going to claim it's impossible to build an AI with any particular set of capabilities. What I'm trying to draw attention to is that knowledge gives human beings a fair amount of leverage. A bunch of companies could decide to pool their knowledge to build an AI that's more powerful than what any of them could build individually, but I think this is a very different scenario from the "fast takeoff" scenario where an AI uses self-improvement to become the most powerful entity in the world practically overnight.
Another important point is that a lot of knowledge is tacit—even the person who has knowledge sometimes finds it difficult to summarize in a form that others will find useful. So I absolutely expect a lot of data sharing in some domains, but I think there will be other domains where incumbent organizations have deep expertise that can't easily be transferred to an AI entity. It's complicated.
Wonderful article! A not insignificant amount of valuable knowledge is also ‘process knowledge’ which is gained via experience in the physical world and interactions with other humans. This is hard to write down, let alone database for an AI to learn.
Yes absolutely. A closely related concept is tacit knowledge.
Really appreciated this article pointing out distinctions that get overlooked when discussing the different kinds of knowledge that AI and humans can acquire. Small note: "Garry Kasparov" is the correct spelling of the former world chess champion. :)
Good catch thank you!