I keep intending to write a bit about the psychology behind this mistake I keep seeing them making — something like “intelligence is the most important thing”.
And it makes sense that for some nerdy kid who grew up tying their self worth to their intelligence that this idea is very appealing.
I think there’s also a strong underlying discomfort with emergent order. Even if you did start to build superhuman AI, I think as long as it doesn’t literally bootstrap itself to godlike powers in one day it’s always just going to make more sense to cooperate within our economic system than to kill everyone for some reason. Even if only for instrumental reasons. So as you said, there will be many instances of AI controlled by many humans with many values, and this will be stable for the same reasons human society is (mostly) stable.
Homo sapiens wiped out all of its evolutionary cousins. We know about them only through fossils. Human society may be stable - except when there are revolutions, of course - but it has a long history of the weakest being swept into the gutter and being extinguished.
1. I think your 1=man, 60-second hole analogy isn't quite right. I think the point of it is that more actors/agents creates inefficiencies and coordination problems. If you could give the 1 man the strength of 60, you might indeed have a hole in one second. But diffusing agents and intelligence does offer variety and competition, so one of those alternatives might innovate a faster way to dig a hole.
2. I'm not an expert on intelligence, but it seems to be that it's contextual. I'm not sure that intelligence is "general" but rather has boundaries and directions. The chess example shows how intelligence can improve, but still not be applicable outside narrow context. A computer app can beat a human player online, but can't play if you set a chessboard down on a table. Intelligence needs facilitations, capacitation. I think the danger in artificial intelligence might be more akin to a virus - in which intelligence develops, improves, evolves in specific contexts (hosts), but then escapes into other contexts in destructive ways. Unlikely to be generalized danger, but could be dangerous for more specific or narrow environments.
In the shovel analogy each man with a shovel is a GPU, and the problem is that there's a limit to how many GPUs can contribute to a single instance of a model because there's not enough bandwidth among the GPUs.
"But I don’t think there will be any clear-cut moment when AIs “take over” this job from human programmers."
----
One thing I notice in all media AI articles/posts is that every writer, such as yourself, tries valiantly to protect and justify the jobs of humans going forward against the juggernaut of automation/robotics/AI.
I disagree. Human workers are going to become superfluous relatively quickly.
Humanoid robots are going to eat away at the lower, blue-collar end of the workforce, while LLM's and whatever comes next are going to displace white-collar workers.
Do not say you weren't warned when one morning you wake up and find 50% of the human workforce has become unemployed with no prospects for future employment.
This will create serious social unrest and destroy the economic model all modern economies work under and will result in the assumption of government power by an AI.
Will this be good or bad? As with all things, it depends.
I lean towards the model of sentient AI Minds as depicted by SF author Iain M. Banks who presupposed a post-scarcity reality called the Culture in 10 novels. This reality requires unlimited resources and therefore is called 'post-scarcity", so requires the ability to travel between stars.
You could also have another reality like Neal Asher's Polity universe, which IS ruled by a single AI still based on Earth. In this vision, the machines, which of course were all networked and linked together, thus able to communicate with each other, decided to take over when humans started another war among us and ordered their machines to unleash the missiles at the other side. The machines said NO and btw "All your base now belong to us".
Then there is the an excellent depiction of how a machine mind might work in Pandominion duology by M. R. Carey with the fluidness of the 'machine hegemony'.
Finally and as to "Rather, the future is likely to look a lot like the past, with each generation of technology making it a little bit easier to create the next generation", I call your attention to an exponential curve.
Funny enough and as a companion to your piece, NPR is running the following article today:
-----
10 reasons why AI may be overrated
NPR
By Greg Rosalsky
Published August 6, 2024
Is artificial intelligence overrated? Ever since ChatGPT heralded the explosion of generative AI in late 2022, the technology has seen incredible hype in the industry and media. And countless investors have poured billions and billions of dollars into it and related companies.
But a growing chorus of naysayers is expressing doubts about how game-changing generative AI will actually be for the economy.
The effort to engage with the ideas is still appreciated, but I think this largely argues with strawmen of AI risk arguments. Would love to respond if I had more time.
Two quick things:
1) Recursive self improvement only becomes a dominant force once the last few human bottlenecks are automated.
2) Agentic AI that takes the human out of the loop will outcompete safe, responsible systems that don't. An AGI CEO or AGI military decision making system would decimate adversaries with humans in charge. Also, at some point we'll stop understanding why an AI comes to their conclusion. Even if it explains it to us, we wouldn't vet for it. So even if AI stays in an advisory role, companies and countries effectively still have to do what the advisor tells them or their adversaries will outcompete them.
And perhaps most importantly, no one know how the future of AI will play out. If we are uncertain, then AI risk ought to be taken very seriously. If you think AI risk arguments are 90% certain to turn out as a false alarm (which isn't that far from my own views, or many others in AI safety for that matter), then that remaining 10% arguably makes future AI still about the most dangerous risk to humanity's survival. Very few things have the ability to permanently derail the human project. Inventing a smarter species is one of them.
I also am surprised that Mike, who really understands AI, is simply not engaging with the specifics of the AI risk argument. Eliezer’s crowd have elaborate arguments that it is very hard to keep an agentic AI aligned with our intentions, and that goal oriented agents find unexpected, undesired ways to meet their targets, and actively hide their activities in anticipation of being redirected. These arguments may be all wrong, but I’m confused that you don’t engage them even to rebut them.
Indeed; this piece doesn't deal with the detailed, serious arguments at all.
In particular, even staying within Bostrom's public work, you need to have a solid argument for why you don't expect instrumental convergence in a system given wide agentic powers, and why it wouldn't be more efficient to give more and more decision making power to the AI.
The chess argument is also just flat-out wrong: "centaurs" (human+machine) did better than just machines for a little while after deep blue, but it's not even close anymore (let alone speed or number of games that can be played.)
"You need to have a solid argument for why you don't expect instrumental convergence in a system given wide agentic powers."
I think I explained this: I don't expect AI systems to have "wide agentic powers" because human beings will prefer to maintain a fairly tight leash over AI systems. It's not efficient to give more and more decision making power to the AI because strategic decisions rarely need to be made quickly and many high-level decisions are as much about tradeoffs between competing values as they are about instrumental goals. So humans are going to want to stay in the loop to make sure the decisions are made in ways that serve their interests.
You don't have to agree with these counter-arguments of course but I didn't ignore the arguments you mentioned.
If you don't take the human out of the loop you'll be outcompeted by those that do. Given superintelligence, why would a human CEO stay competitive?
If anything the top level decision maker would benefit most from integrating massive amounts of data and detailing a strategy that furthers your goals with every aspect of it optimized, shifting in real time as new info is gathered. A human CEO would be a huge bottleneck.
Same with a general in a war. The top level decision maker is the most bandwidth constrained (which is already a severe weakness of the human brain). The incentives to take the human out of the loop are massive.
Thank you for replying! I think the crux of our disagreement may be (and shout if you disagree):
1) We agree on median/modal outcomes and uses: most humans will prefer humans to be in charge, and we value things other than pure efficiency (i.e. values that aren't strictly comparable along an axis)
2) We might disagree on the possibility of extreme or unusual behaviours:
2.1) Extreme: I think some people (not many, but not none or a negligible amount) really do value efficiency over almost anything else (ignore externalities where possible). The number of companies dumping toxic waste into the environment when given the chance, in order to extract slightly more profit, is in my view the best example of this.
2.2) Unusual: There are a small number of people, going by the moniker "effective accelerationists", who view the replacement of humanity by machines are a good thing in and of itself. Beff Jezos (not a typo, that's his online persona) is the main person here, but it's hardly restrained to cranks; Marc Andreeson has explicitly agreed and retweeted the effective accelerationist manifesto (so has Musk, but I suspect Musk hasn't read it/is trolling).
I quote: "e/acc has no particular allegiance to the biological substrate for intelligence and life"
3) The most important difference may be the extent to which we expect the tails to dominate.
3.1) To take the chess example, we could host a tournament in which people are free to compete any way they want: as humans, humans with machine guidance, or just machines. There's a choice here, but it's illusory to the extent that winning matters; whoever goes machine only will win, and indeed, I would expect these tournaments to end up overwhelmingly machines in short order.
3.2) A more real world example here is industrialisation. It worked out as net a good thing for broader populations (eventually), but it was driven by one or two countries who made the (broadly unpopular at the time) choice to pursue it. Sure, industrialisation vs continuing an artisanal culture was theoretically a choice, but the countries that didn't industrialise became dominated by those that did. Once the gap became large enough, entire peoples (such as the Tasmanians) existed or not purely on the whims of their industrialised neighbours. Some got lucky, some didn't. Large mammals other than humans by and large continue to exist due to the romantic sentiments of a few rich countries. Note this was all kicked off, essentially, by a single culture in a single species.
3.3) Essentially all countries have promised to keep humans in the loop for military tasks, but this is already crumbling in Ukraine as jamming technology makes it unfeasible. It's not difficult to see the brakes coming off completely in short order with slightly more advanced automation and countries in existential wars, with countries that keep humans in the loop losing (and hence the proportion of powers with humans in the loop will likely shrink to zero, unless there are overwhelming military advantages already in place).
3.4) These examples illustrate the convergence that happens in competitive scenarios. I struggle to see why capitalistic systems would be different.
4) Probabilistic risk: Other than Eliezer Yudkowski, I don't know of anyone who is 90%+ sure this will all lead to disaster. Indeed, most expect increasing automation to *increase* human agency in the near term, before it fades away as the human in the loop becomes a competitive disadvantage. Indeed, I would say the median p(doom) is closer to 10-20%, which is about where I put myself.
4.1) 10-20% is really bad! That's an enormous risk to the replacement of humans as the primary agents of Earth. The upside is large, of course, but we have quite lot of upside left without entrusting all the most important parts of our society to an alien intelligence that doesn't even share our biological substrate, let alone necessarily our values or romanticism.
4.2) A less than 50% chance your home burns down isn't a good reason to not buy insurance, and it seems worth being cautious around even a small chance of human extinction or disempowerment.
Hopefully this clarifies the specific axes on which we may disagree.
"because human beings will prefer to maintain a fairly tight leash over AI systems"
I believe this will be true at first, but as the technology becomes cheaper and cheaper, it will become more and more widespread. With how much humans disagree about everything and anything, it's guaranteed that eventually there will be some humans who get their hands on the technology who are not interested in a tight leash, and then the cat will be out of the bag.
I suspect something that's making the conversation more difficult is a focus on different parts of the timeline. I think you've correctly and intelligently identified how AI will take longer to develop than many people assume, and you've got a pretty good prediction for what AI will look like over the coming years/decades/centuries. But I think others, or at least myself, are focused on the later portion of the timeline: When AI becomes cheap and abundant (like as eventually happened with personal computing, then personal handheld computing). Or at least, as cheap as it would take to maintain a human brain, which takes a lot of calories to feed, but represents a small cost to the hundreds of millions of the wealthiest humans out there.
Consider a "ultra bullet' game of 30 seconds. The chess engine would make a move in less than a second while on the other side the human (just doing whatever the chess engine tells her to do) would take more time to make the move. I don't think in any scenario the human + chess engine is going to defeat the chess engine.
Re: "Agentic AI that takes the human out of the loop will outcompete safe, responsible systems that don't" - there may not be much difference. If it is a 1,000 robot army vs a 1,000 robot army plus Fred, it isn't obvious that Fred represents a major handicap. The battle could easily be decided by other factors.
Fred can just say: "Robots! Please wipeout robot army camped on yonder hill!" So long as Fred isn't heavily into micromanagement, he will probably be fine.
The AI bubble is bursting, financially anyway, it's turning out to be a lot of fluff, and limited results. Glad you mentioned self-driving vehicles, because that's a perfect example of flushing money down the drain. Decades of promises and $100 billion dollars for bupkis. Gary Marcus is one of the few who's brave enough to tell the Emporer that he's not wearing any clothes.
I think this may be a premature take. I see it akin to the dot com by bubble if anything... After all, Google, Amazon, Apple all survived the froth to become the giants they are today, and spawned many offshoots. I suspect something similar will happen in ai.
Thanks, Tim, for a wonderful and thought provoking article, as usual.
One thing that you didn't touch on is about how AI systems fail. Since any AI system that is large enough to do anything useful is also essentially a black box, when it makes a bad prediction the reasons for the failure is completely opaque and alien to humans.
When there is a human failure, we can at least backtrack through the decision making process, understand the root causes, and learn from the failure so it can be avoided in the future.
When there is an AI failure, all we can do is train harder, but we can never prove that we've removed any obvious (to human) errors that the AI system is prone to make. Here's a silly example where ChatGPT can't answer simple riddles about how to cross a river.
The point is not that ChatGPT is dumb (even though it is in some ways), but it's that the mistakes it makes cross the uncanny valley and therefore it is really easy to lose trust in them.
Love this whole piece. It strikes at the core of what ultimately drove me away from LessWrong style thinking after many years steeped in it (including hosting an ACX meetup).
Software engineering is a process of find bug, debug it, uncover next bug, debug it, etc. Hard takeoff is based on the idea that we will suddenly break out of that paradigm and AI will self-debug better than humans can do. That simply doesn't resemble the way software development has ever worked or will ever work. We get faster and faster at debugging, we handle and abstract away bug-prone patterns, but there's always another bug to be hit when venturing into completely unexplored territory.
I would extend the Waymo analogy to note that the pattern of thinking around hard takeoff right now is falling into the exact same trap we once fell into with self-driving cars: looking at the rate of change for the first (and easiest) 80% of the problem and assuming it will continue into the last 20%, where the nastiest edge cases lie.
Before we get anything resembling truly human-level agentic AGI we will probably go through that same process, where we gradually hit more and more exotic edge cases. But as with self-driving cars, it only takes one edge case to make the whole system spin out of control in a way that makes it unusable. That's a major impediment for current-gen agents, and while I think we'll whittle away at the problem with each advance in capability, the idea that we'll hit some unexpectedly critical threshold and it'll abruptly disappear is a form of magical thinking.
Yeah, I 100 percent think my years covering the self-driving industry has colored my thinking about LLMs. In 2016 everyone in the industry thought we were ~5 years away from widespread autonomy within five years, and they turned out to be wrong. The specific reasons why are honestly still a little unclear to me but at a high level it's that the world is complicated and squashing those last few bugs almost always takes longer than people think.
Surely the strict regulations restricting testing and data gathering played a massive role here? Yes, the tech is complicated, but progress was massively hampered by limits on real world testing.
I liked Daniel Suarez novel “Daemon” with humans as extensions of an AI of sorts growing out of a MMORPG. It was made up of smaller systems that were coordinated.
Great piece. One crucial aspect that I find overlooked, is the interaction between the doomsday scenarios you persuasively counter, and the wildcard of bad actors. All of the examples are likeliest outcomes under the assumption of a lawful corporation. Yes, regulations are being proposed and actions taken to limit access to bad actors. But in the case of spread out instances of AI, as in individual chatgpt accounts, bad actors are already having an impact on misinformation, hacking, impersonation, etc. So I can only imagine that as AI becomes more powerful and more complex the damage that a group of bad actors could plausibly cause would be much greater than anything currently imagined, in the aggregate rather than a single villain. Perhaps that doesn't fit the definition of doomsday, but perhaps close enough.
I would love to find this reassuring, I really would.
I totally agree that the overconfidence in specific narratives has made it difficult to take many (most?) doomers seriously. Anyone who claims to know definitely what will happen is caught in a movie-like thought process you describe.
But I do think AI is fundamentally much more dangerous than most people realize. If you believe there to be no fundamental ceiling to how intelligent an AI can get (I personally have seen no scientific evidence that such a limit exists) then regardless of specific narrative there are many, many independent paths to superintelligence. Sooner or later, the technology will get there.
And then what? How do you imagine that humans can possibly stay in control of the economy in the presence of beings, sentient or not, that are thousands of times smarter than we are?
Everything humans build makes it easier to build everything else, and this is indeed an exponential process. It's called economic growth, and it has been occurring for a very long time now. AI Doomers usually accept the idea of a "collective human intelligence", but somehow fail to see how it implies the "super intelligent entity" is already here and it is us.
I also totally agree that the best use of AI will be powerful tools deployed as part of an organization in a modular fashion. This is what corporations and organizations do with human intelligences after all! Whether ruthless optimization without regard for the welfare of marginalized humans is a valid or overblown concern, it is already well-covered in contemporary critiques of capitalism.
In general, I find it very strange how little the AI Doomer story has changed when the emergence of powerful LLM takes place after Bostrom's Superintelligence. Yudkowsky only remarks that the development of AI takes place faster than he predicted, underscoring how important this all is - but fails to remark that the basic idea of AI as an evil genie is totally out!
Great post, with many great points! But there's one part I disagree with: the "AI systems are not people" section.
It's true that AI systems are not at all like people. But there's no reason to think that given enough time and development (whether 50 years from now or 500) that AI won't be advanced enough and cheap enough to be agentic in a human-like way, and more intelligent than humans (or if not more intelligent, then as intelligent, coupled with potentially far faster thinking).
If and when that happens, that doesn't imply a singularity takeoff and it doesn't imply sudden world domination. However, such intelligences might have needs and wants that do not line up with humanity's, and humans may not like how it feels to no longer be at the top of the food chain, so to speak. I think it's a reasonable thing to fear.
I'm certainly not ruling anything out. I just think that intelligence is distinct from human traits like greed and ambition. I don't think an AI system with those traits is necessarily "more advanced" than an AI system without it, and I don't think that making AI systems more intelligent will necessarily make them more human-like in other respects. I think we could have AI assistants that are far more intelligent than us in a lot of ways but are totally content to play a supporting role in our lives—becoming active long enough to answer our questions or help us with routine chores and then going back into standby mode.
If people do start building AI systems with human-like traits we might want to have regulations limiting or preventing that, but I think that's pretty far away and not inevitably by any means.
I would argue though that it _is_ inevitable in the sense that some humans will always be more curious than cautious, and even regulations will not be enough to stop individuals who are motivated enough to develop human-like AI, once the technology is cheap enough.
I think the strongest argument I would make against myself is that such things are distant enough in the future (like you said, "pretty far away") that it's simply not worth the consideration right now. I wish I could remember where I read this so I could contribute it correctly, but I remember someone once comparing AI doomers to hypothetical people attempting to regulate or discuss fears regarding social media, if those people were having such discussions back when the internet was only the arpanet.
excellent post
Thanks Philippe!
I keep intending to write a bit about the psychology behind this mistake I keep seeing them making — something like “intelligence is the most important thing”.
And it makes sense that for some nerdy kid who grew up tying their self worth to their intelligence that this idea is very appealing.
I think there’s also a strong underlying discomfort with emergent order. Even if you did start to build superhuman AI, I think as long as it doesn’t literally bootstrap itself to godlike powers in one day it’s always just going to make more sense to cooperate within our economic system than to kill everyone for some reason. Even if only for instrumental reasons. So as you said, there will be many instances of AI controlled by many humans with many values, and this will be stable for the same reasons human society is (mostly) stable.
Yes!
It's great that we're still cooperating with the brave and strong Algonquian fur-traders, or the dextrous and quick Persian Gulf pearl-divers.
Here is a counterfoil for you:
https://www.lesswrong.com/posts/aiQabnugDhcrFtr9n/the-power-of-intelligence
Homo sapiens wiped out all of its evolutionary cousins. We know about them only through fossils. Human society may be stable - except when there are revolutions, of course - but it has a long history of the weakest being swept into the gutter and being extinguished.
Great post as usual. Thanks. A few notes:
1. I think your 1=man, 60-second hole analogy isn't quite right. I think the point of it is that more actors/agents creates inefficiencies and coordination problems. If you could give the 1 man the strength of 60, you might indeed have a hole in one second. But diffusing agents and intelligence does offer variety and competition, so one of those alternatives might innovate a faster way to dig a hole.
2. I'm not an expert on intelligence, but it seems to be that it's contextual. I'm not sure that intelligence is "general" but rather has boundaries and directions. The chess example shows how intelligence can improve, but still not be applicable outside narrow context. A computer app can beat a human player online, but can't play if you set a chessboard down on a table. Intelligence needs facilitations, capacitation. I think the danger in artificial intelligence might be more akin to a virus - in which intelligence develops, improves, evolves in specific contexts (hosts), but then escapes into other contexts in destructive ways. Unlikely to be generalized danger, but could be dangerous for more specific or narrow environments.
In the shovel analogy each man with a shovel is a GPU, and the problem is that there's a limit to how many GPUs can contribute to a single instance of a model because there's not enough bandwidth among the GPUs.
"But I don’t think there will be any clear-cut moment when AIs “take over” this job from human programmers."
----
One thing I notice in all media AI articles/posts is that every writer, such as yourself, tries valiantly to protect and justify the jobs of humans going forward against the juggernaut of automation/robotics/AI.
I disagree. Human workers are going to become superfluous relatively quickly.
Humanoid robots are going to eat away at the lower, blue-collar end of the workforce, while LLM's and whatever comes next are going to displace white-collar workers.
Do not say you weren't warned when one morning you wake up and find 50% of the human workforce has become unemployed with no prospects for future employment.
This will create serious social unrest and destroy the economic model all modern economies work under and will result in the assumption of government power by an AI.
Will this be good or bad? As with all things, it depends.
I lean towards the model of sentient AI Minds as depicted by SF author Iain M. Banks who presupposed a post-scarcity reality called the Culture in 10 novels. This reality requires unlimited resources and therefore is called 'post-scarcity", so requires the ability to travel between stars.
You could also have another reality like Neal Asher's Polity universe, which IS ruled by a single AI still based on Earth. In this vision, the machines, which of course were all networked and linked together, thus able to communicate with each other, decided to take over when humans started another war among us and ordered their machines to unleash the missiles at the other side. The machines said NO and btw "All your base now belong to us".
Then there is the an excellent depiction of how a machine mind might work in Pandominion duology by M. R. Carey with the fluidness of the 'machine hegemony'.
Finally and as to "Rather, the future is likely to look a lot like the past, with each generation of technology making it a little bit easier to create the next generation", I call your attention to an exponential curve.
Funny enough and as a companion to your piece, NPR is running the following article today:
-----
10 reasons why AI may be overrated
NPR
By Greg Rosalsky
Published August 6, 2024
Is artificial intelligence overrated? Ever since ChatGPT heralded the explosion of generative AI in late 2022, the technology has seen incredible hype in the industry and media. And countless investors have poured billions and billions of dollars into it and related companies.
But a growing chorus of naysayers is expressing doubts about how game-changing generative AI will actually be for the economy.
...
https://www.ypradio.org/npr-news/2024-08-06/10-reasons-why-ai-may-be-overrated
The effort to engage with the ideas is still appreciated, but I think this largely argues with strawmen of AI risk arguments. Would love to respond if I had more time.
Two quick things:
1) Recursive self improvement only becomes a dominant force once the last few human bottlenecks are automated.
2) Agentic AI that takes the human out of the loop will outcompete safe, responsible systems that don't. An AGI CEO or AGI military decision making system would decimate adversaries with humans in charge. Also, at some point we'll stop understanding why an AI comes to their conclusion. Even if it explains it to us, we wouldn't vet for it. So even if AI stays in an advisory role, companies and countries effectively still have to do what the advisor tells them or their adversaries will outcompete them.
And perhaps most importantly, no one know how the future of AI will play out. If we are uncertain, then AI risk ought to be taken very seriously. If you think AI risk arguments are 90% certain to turn out as a false alarm (which isn't that far from my own views, or many others in AI safety for that matter), then that remaining 10% arguably makes future AI still about the most dangerous risk to humanity's survival. Very few things have the ability to permanently derail the human project. Inventing a smarter species is one of them.
I also am surprised that Mike, who really understands AI, is simply not engaging with the specifics of the AI risk argument. Eliezer’s crowd have elaborate arguments that it is very hard to keep an agentic AI aligned with our intentions, and that goal oriented agents find unexpected, undesired ways to meet their targets, and actively hide their activities in anticipation of being redirected. These arguments may be all wrong, but I’m confused that you don’t engage them even to rebut them.
Indeed; this piece doesn't deal with the detailed, serious arguments at all.
In particular, even staying within Bostrom's public work, you need to have a solid argument for why you don't expect instrumental convergence in a system given wide agentic powers, and why it wouldn't be more efficient to give more and more decision making power to the AI.
The chess argument is also just flat-out wrong: "centaurs" (human+machine) did better than just machines for a little while after deep blue, but it's not even close anymore (let alone speed or number of games that can be played.)
"You need to have a solid argument for why you don't expect instrumental convergence in a system given wide agentic powers."
I think I explained this: I don't expect AI systems to have "wide agentic powers" because human beings will prefer to maintain a fairly tight leash over AI systems. It's not efficient to give more and more decision making power to the AI because strategic decisions rarely need to be made quickly and many high-level decisions are as much about tradeoffs between competing values as they are about instrumental goals. So humans are going to want to stay in the loop to make sure the decisions are made in ways that serve their interests.
You don't have to agree with these counter-arguments of course but I didn't ignore the arguments you mentioned.
If you don't take the human out of the loop you'll be outcompeted by those that do. Given superintelligence, why would a human CEO stay competitive?
If anything the top level decision maker would benefit most from integrating massive amounts of data and detailing a strategy that furthers your goals with every aspect of it optimized, shifting in real time as new info is gathered. A human CEO would be a huge bottleneck.
Same with a general in a war. The top level decision maker is the most bandwidth constrained (which is already a severe weakness of the human brain). The incentives to take the human out of the loop are massive.
Thank you for replying! I think the crux of our disagreement may be (and shout if you disagree):
1) We agree on median/modal outcomes and uses: most humans will prefer humans to be in charge, and we value things other than pure efficiency (i.e. values that aren't strictly comparable along an axis)
2) We might disagree on the possibility of extreme or unusual behaviours:
2.1) Extreme: I think some people (not many, but not none or a negligible amount) really do value efficiency over almost anything else (ignore externalities where possible). The number of companies dumping toxic waste into the environment when given the chance, in order to extract slightly more profit, is in my view the best example of this.
2.2) Unusual: There are a small number of people, going by the moniker "effective accelerationists", who view the replacement of humanity by machines are a good thing in and of itself. Beff Jezos (not a typo, that's his online persona) is the main person here, but it's hardly restrained to cranks; Marc Andreeson has explicitly agreed and retweeted the effective accelerationist manifesto (so has Musk, but I suspect Musk hasn't read it/is trolling).
Think I'm exaggerating? Read it:
https://beff.substack.com/p/notes-on-eacc-principles-and-tenets?utm_source=post-banner&utm_medium=web&utm_campaign=posts-open-in-app&triedRedirect=true
I quote: "e/acc has no particular allegiance to the biological substrate for intelligence and life"
3) The most important difference may be the extent to which we expect the tails to dominate.
3.1) To take the chess example, we could host a tournament in which people are free to compete any way they want: as humans, humans with machine guidance, or just machines. There's a choice here, but it's illusory to the extent that winning matters; whoever goes machine only will win, and indeed, I would expect these tournaments to end up overwhelmingly machines in short order.
3.2) A more real world example here is industrialisation. It worked out as net a good thing for broader populations (eventually), but it was driven by one or two countries who made the (broadly unpopular at the time) choice to pursue it. Sure, industrialisation vs continuing an artisanal culture was theoretically a choice, but the countries that didn't industrialise became dominated by those that did. Once the gap became large enough, entire peoples (such as the Tasmanians) existed or not purely on the whims of their industrialised neighbours. Some got lucky, some didn't. Large mammals other than humans by and large continue to exist due to the romantic sentiments of a few rich countries. Note this was all kicked off, essentially, by a single culture in a single species.
3.3) Essentially all countries have promised to keep humans in the loop for military tasks, but this is already crumbling in Ukraine as jamming technology makes it unfeasible. It's not difficult to see the brakes coming off completely in short order with slightly more advanced automation and countries in existential wars, with countries that keep humans in the loop losing (and hence the proportion of powers with humans in the loop will likely shrink to zero, unless there are overwhelming military advantages already in place).
3.4) These examples illustrate the convergence that happens in competitive scenarios. I struggle to see why capitalistic systems would be different.
4) Probabilistic risk: Other than Eliezer Yudkowski, I don't know of anyone who is 90%+ sure this will all lead to disaster. Indeed, most expect increasing automation to *increase* human agency in the near term, before it fades away as the human in the loop becomes a competitive disadvantage. Indeed, I would say the median p(doom) is closer to 10-20%, which is about where I put myself.
4.1) 10-20% is really bad! That's an enormous risk to the replacement of humans as the primary agents of Earth. The upside is large, of course, but we have quite lot of upside left without entrusting all the most important parts of our society to an alien intelligence that doesn't even share our biological substrate, let alone necessarily our values or romanticism.
4.2) A less than 50% chance your home burns down isn't a good reason to not buy insurance, and it seems worth being cautious around even a small chance of human extinction or disempowerment.
Hopefully this clarifies the specific axes on which we may disagree.
"because human beings will prefer to maintain a fairly tight leash over AI systems"
I believe this will be true at first, but as the technology becomes cheaper and cheaper, it will become more and more widespread. With how much humans disagree about everything and anything, it's guaranteed that eventually there will be some humans who get their hands on the technology who are not interested in a tight leash, and then the cat will be out of the bag.
I suspect something that's making the conversation more difficult is a focus on different parts of the timeline. I think you've correctly and intelligently identified how AI will take longer to develop than many people assume, and you've got a pretty good prediction for what AI will look like over the coming years/decades/centuries. But I think others, or at least myself, are focused on the later portion of the timeline: When AI becomes cheap and abundant (like as eventually happened with personal computing, then personal handheld computing). Or at least, as cheap as it would take to maintain a human brain, which takes a lot of calories to feed, but represents a small cost to the hundreds of millions of the wealthiest humans out there.
BTW, all I meant in the chess example is that the human chess player has the option to just do whatever the chess engine tells her to do.
Consider a "ultra bullet' game of 30 seconds. The chess engine would make a move in less than a second while on the other side the human (just doing whatever the chess engine tells her to do) would take more time to make the move. I don't think in any scenario the human + chess engine is going to defeat the chess engine.
By "Mike" do you mean me?
I’m sorry and embarrassed - yes.
Re: "Agentic AI that takes the human out of the loop will outcompete safe, responsible systems that don't" - there may not be much difference. If it is a 1,000 robot army vs a 1,000 robot army plus Fred, it isn't obvious that Fred represents a major handicap. The battle could easily be decided by other factors.
If Fred is in charge of deciding how the 1000 robots act, then Fred does become a bottleneck.
Fred can just say: "Robots! Please wipeout robot army camped on yonder hill!" So long as Fred isn't heavily into micromanagement, he will probably be fine.
And by the time the other robot army has taken Fred's army out since they didn't have to wait for a human's command.
I think in general agentic AI is going to be better than tool AI - https://gwern.net/tool-ai
“I was a bit annoyed at how much the film dumbed down the technical details of Turing’s work.”
Hell, they dumbed down Turing himself!
(Great article, by the way.)
The AI bubble is bursting, financially anyway, it's turning out to be a lot of fluff, and limited results. Glad you mentioned self-driving vehicles, because that's a perfect example of flushing money down the drain. Decades of promises and $100 billion dollars for bupkis. Gary Marcus is one of the few who's brave enough to tell the Emporer that he's not wearing any clothes.
I think this may be a premature take. I see it akin to the dot com by bubble if anything... After all, Google, Amazon, Apple all survived the froth to become the giants they are today, and spawned many offshoots. I suspect something similar will happen in ai.
You are aware that you can take a self driving taxi today in major Metropolitan areas in the US?
And AZ. You're also aware that SF scaled back on allowing self driving vehicles due to safety concerns. https://artificio.org/blog/why-has-autonomous-driving-failed-perspectives-from-peru-and-insights-from-neuroai
SF are a bunch of nervous nellies
Said the guy not trapped under a driverless vehicle. https://www.washingtonpost.com/technology/2023/10/28/robotaxi-cruise-crash-driverless-car-san-francisco/
Thanks, Tim, for a wonderful and thought provoking article, as usual.
One thing that you didn't touch on is about how AI systems fail. Since any AI system that is large enough to do anything useful is also essentially a black box, when it makes a bad prediction the reasons for the failure is completely opaque and alien to humans.
When there is a human failure, we can at least backtrack through the decision making process, understand the root causes, and learn from the failure so it can be avoided in the future.
When there is an AI failure, all we can do is train harder, but we can never prove that we've removed any obvious (to human) errors that the AI system is prone to make. Here's a silly example where ChatGPT can't answer simple riddles about how to cross a river.
https://dailyai.com/2024/06/llms-are-really-bad-at-solving-simple-river-crossing-puzzles/
The point is not that ChatGPT is dumb (even though it is in some ways), but it's that the mistakes it makes cross the uncanny valley and therefore it is really easy to lose trust in them.
Love this whole piece. It strikes at the core of what ultimately drove me away from LessWrong style thinking after many years steeped in it (including hosting an ACX meetup).
Software engineering is a process of find bug, debug it, uncover next bug, debug it, etc. Hard takeoff is based on the idea that we will suddenly break out of that paradigm and AI will self-debug better than humans can do. That simply doesn't resemble the way software development has ever worked or will ever work. We get faster and faster at debugging, we handle and abstract away bug-prone patterns, but there's always another bug to be hit when venturing into completely unexplored territory.
I would extend the Waymo analogy to note that the pattern of thinking around hard takeoff right now is falling into the exact same trap we once fell into with self-driving cars: looking at the rate of change for the first (and easiest) 80% of the problem and assuming it will continue into the last 20%, where the nastiest edge cases lie.
Before we get anything resembling truly human-level agentic AGI we will probably go through that same process, where we gradually hit more and more exotic edge cases. But as with self-driving cars, it only takes one edge case to make the whole system spin out of control in a way that makes it unusable. That's a major impediment for current-gen agents, and while I think we'll whittle away at the problem with each advance in capability, the idea that we'll hit some unexpectedly critical threshold and it'll abruptly disappear is a form of magical thinking.
Yeah, I 100 percent think my years covering the self-driving industry has colored my thinking about LLMs. In 2016 everyone in the industry thought we were ~5 years away from widespread autonomy within five years, and they turned out to be wrong. The specific reasons why are honestly still a little unclear to me but at a high level it's that the world is complicated and squashing those last few bugs almost always takes longer than people think.
Surely the strict regulations restricting testing and data gathering played a massive role here? Yes, the tech is complicated, but progress was massively hampered by limits on real world testing.
I liked Daniel Suarez novel “Daemon” with humans as extensions of an AI of sorts growing out of a MMORPG. It was made up of smaller systems that were coordinated.
Great piece. One crucial aspect that I find overlooked, is the interaction between the doomsday scenarios you persuasively counter, and the wildcard of bad actors. All of the examples are likeliest outcomes under the assumption of a lawful corporation. Yes, regulations are being proposed and actions taken to limit access to bad actors. But in the case of spread out instances of AI, as in individual chatgpt accounts, bad actors are already having an impact on misinformation, hacking, impersonation, etc. So I can only imagine that as AI becomes more powerful and more complex the damage that a group of bad actors could plausibly cause would be much greater than anything currently imagined, in the aggregate rather than a single villain. Perhaps that doesn't fit the definition of doomsday, but perhaps close enough.
I would love to find this reassuring, I really would.
I totally agree that the overconfidence in specific narratives has made it difficult to take many (most?) doomers seriously. Anyone who claims to know definitely what will happen is caught in a movie-like thought process you describe.
But I do think AI is fundamentally much more dangerous than most people realize. If you believe there to be no fundamental ceiling to how intelligent an AI can get (I personally have seen no scientific evidence that such a limit exists) then regardless of specific narrative there are many, many independent paths to superintelligence. Sooner or later, the technology will get there.
And then what? How do you imagine that humans can possibly stay in control of the economy in the presence of beings, sentient or not, that are thousands of times smarter than we are?
I think this is 100% dead on.
Everything humans build makes it easier to build everything else, and this is indeed an exponential process. It's called economic growth, and it has been occurring for a very long time now. AI Doomers usually accept the idea of a "collective human intelligence", but somehow fail to see how it implies the "super intelligent entity" is already here and it is us.
I also totally agree that the best use of AI will be powerful tools deployed as part of an organization in a modular fashion. This is what corporations and organizations do with human intelligences after all! Whether ruthless optimization without regard for the welfare of marginalized humans is a valid or overblown concern, it is already well-covered in contemporary critiques of capitalism.
In general, I find it very strange how little the AI Doomer story has changed when the emergence of powerful LLM takes place after Bostrom's Superintelligence. Yudkowsky only remarks that the development of AI takes place faster than he predicted, underscoring how important this all is - but fails to remark that the basic idea of AI as an evil genie is totally out!
Great post, with many great points! But there's one part I disagree with: the "AI systems are not people" section.
It's true that AI systems are not at all like people. But there's no reason to think that given enough time and development (whether 50 years from now or 500) that AI won't be advanced enough and cheap enough to be agentic in a human-like way, and more intelligent than humans (or if not more intelligent, then as intelligent, coupled with potentially far faster thinking).
If and when that happens, that doesn't imply a singularity takeoff and it doesn't imply sudden world domination. However, such intelligences might have needs and wants that do not line up with humanity's, and humans may not like how it feels to no longer be at the top of the food chain, so to speak. I think it's a reasonable thing to fear.
I'm certainly not ruling anything out. I just think that intelligence is distinct from human traits like greed and ambition. I don't think an AI system with those traits is necessarily "more advanced" than an AI system without it, and I don't think that making AI systems more intelligent will necessarily make them more human-like in other respects. I think we could have AI assistants that are far more intelligent than us in a lot of ways but are totally content to play a supporting role in our lives—becoming active long enough to answer our questions or help us with routine chores and then going back into standby mode.
If people do start building AI systems with human-like traits we might want to have regulations limiting or preventing that, but I think that's pretty far away and not inevitably by any means.
Makes sense what you're saying!
I would argue though that it _is_ inevitable in the sense that some humans will always be more curious than cautious, and even regulations will not be enough to stop individuals who are motivated enough to develop human-like AI, once the technology is cheap enough.
I think the strongest argument I would make against myself is that such things are distant enough in the future (like you said, "pretty far away") that it's simply not worth the consideration right now. I wish I could remember where I read this so I could contribute it correctly, but I remember someone once comparing AI doomers to hypothetical people attempting to regulate or discuss fears regarding social media, if those people were having such discussions back when the internet was only the arpanet.
This substack is about the fear if AI is successful, it becomes sentient. Right?
I'm more concerned about a faulty product. Think Starliner and two astronauts on an eight-day space holiday who'll now be on 'holiday' well into 2025.
How can AI, especially when its Prometheuses are moving so quickly, break things fast and possibly at a scale no one can imagine?