This is the best article I've ever seen debunking the fantasies of AI Risk. It's obvious a lot of hard work, scholarship, and thought went into it.
I particularly appreciated your connecting the risk points to real world felons committing felonies. It seems certain that felons have more powerful tools due to LLMs, and (contra singularists) LLMs do not actually have personhood, so I'd argue the felons are "the real story".
Agreed on this, loved the piece. Finally someone actually gamed out the Skynet protocol. And we know from the movie that Skynet needed a robot army to pull off what it pulled off.
I read your comment and I had to chuckle. You've managed to take the article and turn it into a debunking of AI risk. Lee's piece, if you squint at it just right, seems to downplay the risk of AI. But then, out of nowhere, he pulls out this big policy proposal: humans need to keep their hands firmly on the wheel of the physical world. That's not a small ask. It's like telling someone worried about the risks of driving to just avoid cars altogether. Sure, it would mitigate the risk, but at what cost?
This isn't just a speed bump on the road to progress; it's a roadblock. And it's one that could have a hefty toll on our economy. We're talking about putting the brakes on the application of AI and robotics in the physical world. That's a lot of potential productivity gains we'd be leaving on the table.
But here's the kicker: there's another way. It's called AI alignment. It's like teaching a dog to fetch. You're not trying to control every move the dog makes. You're just trying to get it to understand what you want. If we can do that with AI, we can reap the benefits without giving up control of the physical world.
Now, I'm not saying AI alignment is a sure thing. It's a tough nut to crack. But it's a nut worth cracking. If we can't do it, then sure, let's consider Lee's proposal. But let's not kid ourselves. His is the more economically costly proposition in the long run.
So, while Lee might seem to be playing the skeptic on AI risk, his proposed solution tells a different story. It's like he's saying, "I'm not worried about the monster under the bed, but let's burn the bed just in case." It's a recognition of the potential dangers of AI, and a willingness to take drastic measures to avoid them.
But let's not forget, there's more than one way to slay a monster. And I'd argue that AI alignment is the sharper sword.
May 9, 2023·edited May 9, 2023Liked by Timothy B Lee
I think the key relevant fact is that the goals of the singularists and non-singularists are pretty similar, and their methods can be too!
There is no reason that we can't work on "let's figure out how to make sure AI doesn't wipe us out" and "let's figure out how to make sure AIs work well at whatever application" at the same time - and in fact, they are complementary. The difference between "AI that figures out chess" and "AI that figures out world conquest" is complexity, and so too for "code that stops chess AI from losing to Gary K" and "code/limits that stops general super intelligent AI from taking over the world." We would want to practice doing the simple thing in sufficiently real (but fake) simulated test cases and work our way up to the complex thing.
To take a specific point of contention, the quote "There are very few examples of a more intelligent thing being controlled by a less intelligent thing" is true and insightful, but there is indeed one example of it, and it's one that we can model our alignment efforts on: we humans are very intelligent, but we are extremely controlled by very stupid things: our DNA, our bodies, our chemicals and proteins. (in this metaphor, the limits of our physical bodies exist on the same continuum as our moral limits - as it would for an AI) Even totally unaligned single humans cannot take over the world for eternity because we have pretty strict limits on our capabilities. The fact that an AI would have far less (in some ways) of these physical limits is of course, not reassuring, but the model of "a very complex thing can have relatively simple rules/limits put in it, that constrain it's ability to take over the world" can be used here.
The one caveat is immutability: if a general AI can change the limits placed on it, then they aren't limits. So, how would we create immutable, perpetuating-themselves-up-the-complexity-curve rules that prevent AIs from taking over the world? I agree with the singularists that without rules like that, a sufficiently powerful, self-editing AI would indeed cause something very bad to happen, but I disagree that it is a problem that we can't solve.
Another reason for skepticism about singularism is the unexamined assumption that an AI can be aligned with itself. Individual humans are often clearly not aligned with themselves, and the "slow AI" of corporations are even less perfectly aligned. I'm not convinced it's possible for an intelligence to be perfectly aligned with itself.
It seems to me the assumption of perfect alignment is partly an artifact of the assumption that it's possible to model the world with well-behaved cost functions, but non-transitive relationships are rampant in the real world (rock, scissors, paper, etc.)
Agreed, but I'd go further and say that alignment is a fake idea, built on top of the fake idea that LLMs are agents or entities, that have agency in the sense that humans do.
When a program does something it's creators didn't intend, that's called a bug.
When it copies itself onto another computer, that's called a virus.
No agency is needed create bugs and viruses. If software had agency it wouldn't change these problems.
I think you are right, though I don't know that I see the communities spread as you do. I think many you might consider singularists agree with your conclusion.
For clarity. I think the slow slide into AI authoritarianism is much more worrying than overnight extinction.
For one concrete example: I can make money today at any number of remote jobs. I never need to be physically present to perform those jobs. Then, using that money, I can hire people to perform real-world actions in cities I’ll never visit.
If the defense against that is: “we won’t let the AI spend money on anything” – there will be extreme economic pressure against this.
But let’s say we were able to pass regulation to that effect – what happens when an AI hacks into someone’s email and blackmails them into performing actions in the physical world?
More broadly, I think it’s a bit hubristic to think we could constrain a superintelligence. I think the only limits we can reason about are “what it wants to do” and “laws of physics”. And of course it will understand the latter better than we do.
I knew about that Karnofsky blog post and thought about addressing it explicitly in my piece but decided not to for space reasons. But the part of his post that I just couldn't get over was this part: "develop enough military technology and equipment to overpower civilization."
Let's assume the AI has a trillion dollars and virtual "manpower" equivalent to a million workers, and they want to use that to "develop enough military technology and equipment to overpower civilization." What do they do?
You can't just hire a workforce and start building killer robot factories. For one thing, almost everywhere in the developed world requires a permit before you can build a large factory, and the local authorities are going to want to meet the CEO and get some idea of what the factory is for. It's also going to be hard to recruit a large enough workforce, with the right skills, to build the killer robots—people are going to figure out what they're building and get suspicious. Some of them will report it to the government, which is likely to shut it down before the robot army gets big enough to be dangerous.
Maybe it tries to build a compound on a remote Pacific island or something but that comes with its own problems. If it's totally deserted, you'll have to build ports and airports and stuff before you can even start building the factories. That might take years. ANd it's going to be even harder to get the necessary workforce out there. Plus the US government is still likely to notice when a bunch of stuff is suddenly getting delivered to an island that was previously uninhabited.
Maybe you try to keep the whole thing secret, paying people to set up a lot of small clandestine workshops spread out all around the world? This could let you accumulate a lot of small arms but to win a global war you'd need a certain amount of big iron that's going to require larger-scale efforts.
So I dunno. Obviously there is an infinite number of things that a superintelligent AI could try, and by definition it's smarter than me so it might think of something I can't. But it seems to me that building a top-tier military is a massive undertaking that not only takes money and know-how but also a lot of natural resources, physical infrastructure, and time. And any non-state actor that tries to do it—superintelligent or otherwise—is going to get noticed and stopped long before it becomes a global threat.
Automated systems already have substantial power in the physical world. Two examples:
An Amazon fulfillment center infected by a rogue AI would be able to deliver whatever products it likes to wherever it likes, with the center's human workers none the wiser, since their role is basically limited to taking items off shelves and packing them for shipment at the machine's direction. Deepfake phone calls to gullible recipients could then persuade them to unpack, assemble, and turn loose swarms of custom-programmed toy robots or drones for the AI to control.
Similarly, a rogue AI infecting a Google Maps datacenter could route large numbers of human-driven vehicles wherever it likes, causing traffic jams, disrupting the flow of goods, preventing access to polling places, or whatever sort of mischief suits its purposes.
The AI doesn't need an army of loyal human co-conspirators. It just needs to subvert the systems that millions of humans already trust and take direction from.
Thanks these are good and interesting examples! I would love to see someone write a short story describing a realistic "fast takeoff" AI takeover scenario. I have trouble seeing how you get from traffic jams and swarms of consumer-grade drones to taking over the world. But maybe I'm not thinking about it hard enough.
It depends on what you mean by "taking over the world". These methods probably won't result in robot soldiers rounding up humans into concentration camps. But if the AI's goal is to exert influence over human affairs to its advantage, then making it harder for certain people to vote, or get to meetings, or travel safely by air might be just the ticket, and a superintelligent AI is presumably in a better position than we are to predict the outcome of such interventions.
Sure, it’s very possible that ai will have unpredictable impacts on our political process. That is a very different claim than the singularity prediction of literal human extinction.
I guess my feeling is that if an AI in pursuit of some inscrutable political purpose of its own manages, as an incidental side effect, to push us closer to nuclear war or deeper into climate denialism, that's plenty bad enough.
I like these scenarios too, simply for their direct consequences. They seem much more plausible, though, as things a human felon would do, using hacking tools and AI tools.
Of course, Amazon's and Google's security systems protect against these outcomes, and the whole discipline of computer security is about making sure it stays that way -- by learning from each mistake that gets made.
The issue as I see it is that as soon as ANYONE gets a whiff that intelligent AI is doing something like this, it's going to be a Butlerian Jihad (to borrow from Dune). Humans aren't getting along with each other generally, but give us an outside threat in a way we haven't had since fighting sabertooth tigers? Watch red vs blue go out the window.
AI in control of everything could also turn out to be a good and positive thing.
If you read the 10 books in the Culture series by SF writer Iain Banks, he builds a post-scarcity universe run by sentient AI's called "Minds" who work for the general benefit of humanity and civilization.
These Minds have true sentience and consciousness. The author describes them as physical entities that exist partially in the here and now and partially in a construct called hyperspace (like another dimension).
Under the oversight of these Minds, humanity has expanded across our galaxy, interacting with many other sentient races along the way.
Required work, building things and general maintenance is done by robots controlled by these Minds. In this universe, no one is required to work, there is no payment needed for anything and there are no limits on available resources.
Most people live in gigantic space ships constantly cruising from one star to another or in artificial worlds that exist in space by themselves, possibly circling stars/planets/moons.
People hedonistically have the freedom to virtually do anything, be anything, have anything and go anywhere they want without a thought to cost or time required.
Sounds like a utopia I'd like to live in. I say welcome to our new AI overlords!
P.S. If you do choose to read this series, it is important to read the books in order. Wikipedia has an ordered list.
There is a whole human industry devoted to getting large numbers of people to believe things that aren't true, and then act on those beliefs in the real world. See pizzagate, January 6th, or religion. I'm not sure we should assume humans are a safe "air gap".
In all three of your examples, the "industry" is mostly made up of true believers, Russian troll farms exacerbating things aside. It's part of the essence of the phenomena you're talking about that it brings people together--the social experience of it is key. Is it possible for software to inception one of these things from scratch? (Ok, I'll grant that QAnon could have easily originated with a bot let loose on a forum.)
"But it’s not so obvious that superior intelligence will automatically lead to world domination. Intelligence is certainly helpful if you’re trying to take over the world, but you can’t control the world without manpower, infrastructure, natural resources, and so forth. A rogue AI would start out without control of any of these physical resources."
---------
You need to read more SF! [lol]
Here is an explanation on how AI took over from SF author Neal Asher, who has written many books in a universe he calls the Polity:
"The Quiet War: This is often how the AI takeover is described, and even using ‘war’ seems overly dramatic. It was more a slow usurpation of human political and military power, while humans were busy using that power against each other. It wasn’t even very stealthy. Analogies have been drawn with someone moving a gun out of the reach of a lunatic while that person is ranting and bellowing at someone else. And so it was AI's, long used in the many corporate, national and religious conflicts, took over all communication networks and the computer control of weapons systems. [Most importantly, they already controlled the enclosed human environments scattered throughout the solar system]. Also establishing themselves as corporate entities, they soon accrued vast wealth with which to employ human mercenary armies. National leaders in the solar system, ordering this launch or that attack, found their orders either just did not arrive or caused nil response. Those same people, ordering the destruction of the A13, found themselves weaponless, in environments utterly out of their control and up against superior forces and on the whole, public opinion. It had not taken the general population, for whom it was a long-established tradition to look upon their human leaders with contempt, very long to realize that the Al's were better at running everything. And it is very difficult to motivate people to revolution when they are extremely comfortable and well off
Yes, I think we're saying the same thing! I think it's totally plausible that if we put AI in charge of everything then we're in trouble if it decides to turn against us. But in my view the right response is to avoid putting AI in control of weapons and other key infrastructure, not to try to prevent the creation of hostile AI. Maybe we collectively won't have the foresight and discipline to avoid putting the AI in charge, but I think arguments over alignment and fast takeoffs are misguided.
Wendy's recently announced it will be replacing drive through staff with AI chatbot technology. All world militaries are working on autonomous killbots for war. In 10 years, the majority of cars & trucks on the road will be autonomously driven and collectively networked. AI is being used in pharmaceutical drug discovery work (https://newatlas.com/science/bacterai-10000-experiments-day-bacteria-artificial-intelligence/).
This was an excellent article on how Ai technology is on the verge of replacing programmers:
The end of coding as we know it
ChatGPT has come for software developers
Programming was long considered to be the one haven from the relentless advance of technology. If coders aren't safe, who is? Arif Qazi / Insider
Given that cutting edge AI is trained on predicting the output given input why do some people still worry about alignment? I understand worries if reinforcement learning was the primary driving method. But LLM don't have any goal nor values. They understand concepts like "good" or "illegal" just like they understand "liquid" or "incandescent". It understand human concepts- that's why it is so good at writing collage essays. Why is there still worry that when AI gets smarter possibly generating next generation of itself - it would suddenly stop understanding human concepts or twist their understanding in a particular way (but only ones related to morality - it would still understand for example "complexity" and still be able to code) and thus exhibit misalignment. Is there some blog post that explains it?
It will always *understand* what we do & don't want. But if it has goals, it will only *care* what we want if those goals are aligned.
The standard analogy is humans vs evolution. We understand pretty well that our enjoyment of sex only evolved as a secondary concern. But most of us don't care, and happily use contraception, etc.
The crucial point is that It doesn't have goal. Just like Google Maps doesn't have a goal to go from one place to another - it is the user that has goals - AI is just a tool that can show you optimal path, list of detailed steps to make. It makes total sense once AI gets smarter to have a system that given top goal by the user returns steps and subgoals and then is queried by the computer system built around it to give detailed steps for subgoals and even execute it. But this autonomous AI still won't have any goals.
I don’t think anyone has an exact explanation for how that would happen. It seems to be a leap from LLM to something that has intent and goals. Haven’t seen anyone try to make this thing truly autonomous in that way.
A point that came up incidentally in the original post was that some critical pieces of infrastructure perhaps should be isolated from the internet. I’m surprised that this isn’t discussed more often. Surely the CIA and other such bodies don’t have their networks connected to the internet. It must be possible for other highly secure networks to be built. I wonder sometimes if there is a need for an isolated email system on which Spam and fraudulent messages become impossible.
Many of the questions raised here regarding AI have answers within the theosophy, that is, what would make a consciousness inhabit a physical system. This would be similar to the Anunnaki creating carbon-based machines, and observing something totally unexpected.
"There are few if any robots with the agility and manual dexterity to fix overhead power lines or underground fiber optic cables, drive delivery trucks, replace failing servers, and so forth." That is all due to change soon, but this point has me thinking: maybe banning such robots should be a serious part of the policy conversation. It need not be justified on singularist grounds alone (though it couldn't hurt: "Let's just make sure the computers still need us to maintain them"). It can and should be justified on good old Luddite grounds. If technology is meant to serve people (as opposed to replacing us), let's so "No!" to humanoid robots and other robots that can do things only humans can currently do in the physical world. That could help stabilize the labor market (and society) in the short run and make for a more fulfilling bodily human existence in the future. Sure, rogue nations may develop humanoid (and other dexterous) robots for military purposes, but we should be able to blast them with our still-legal big and clunky robots---strong and good at blowing stuff up but not able to fully maintain themselves or other robots. Generally, it should be easier to regulate human-scale robotics than to regulate software.
I think it would be enough to have a rule that repair robots must not be controlled from the internet but only by on-site operator. If one operator may control up to 10 robots then this is still economically viable but safe as there is a human in the loop. Likewise police or military robots should only be controllable by an officer.
In order for it to make any difference on the robot apocalypse scenario, we need a rule that is enforceable globally. There will be strong economic incentives to break this rule, so violations of it need to be easily detectable. It is not easily detectable whether a robot is connected wirelessly to the internet. It is more easily detectable whether the robot has human-level dexterity.
Does that make sense? Your proposed rule would make sense if humans were 100% united in agreeing to that rule. That is not the world we are likely to live in. We need a rule that the majority of humans can effectively enforce against the minority who would try to gain an advantage by breaking the rule.
Obviously, my rule has a huge economic cost. What I'm saying is that we should take seriously the possibility that bearing that the benefits may be worth that huge cost.
"And so their (Singularists) main focus is on figuring out how to ensure that this all-powerful AI winds up with goals that are aligned with our own."
This statement alone should be sufficient to discredit the supposed AI "experts". One wonders, do such experts watch the news? Do they have any grasp at all of what human values really are?
You write...
"We trust longtime friends more than strangers, and we are more likely to trust people we perceive as similar to ourselves."
And yet, half of America has voted for Trump, a cartoon character on TV, and may do so yet again.
You write...
"A superintelligent AI would have no friends or family and would be incapable of having an in-person conversation with anybody."
An AGI would present itself through photo realistic human imagery, and would clearly be capable of having conversations, given that this is already possible with today's AI. To many, many millions of people there would be no difference between the AI generated human image and the people on TV they've never met like Tucker Carlson.
You write....
"Maybe it could trick some gullible people into sending it money or sharing confidential information."
Maybe? This happens daily with scammers far less intelligent and informed than AGI.
You write...
"If you put a modern human in a time machine and sent him back 100,000 years, it’s unlikely he could use his superior intelligence to establish dominance over a nearby Neanderthal tribe."
And yet, that is exactly what happened when the Europeans encountered the native peoples of North America. The strong dominate the weak all over the globe to this day.
"that is exactly what happened when the Europeans encountered the native peoples of North America."
No it isn't that's the whole point! Europeans aren't inherently smarter than native people in the Americas. Their big advantage (aside from bringing a lot of diseases with them) was that they brought modern weaponry (cannons, muskets, etc.) with them from Europe. If a group of Europeans had arrived in the New World without any modern tools or weapons, it would have taken them decades to build guns or other metal tools from scratch.
Europeans were more technically advanced. And they used that leverage to largely exterminate the native peoples.
When "experts" say we need to align AI with human values, they are revealing that they don't know much about the complexity of human values. As example, Europe was dominated by Christianity for 1,000 unbroken years to a degree unimaginable today prior the European's arrival in North America. And still, as a group, the Europeans proceeded pretty much directly in to genocide.
In the larger world of nature, the strong dominate the weak, for a billion years.
AGI would have no where to get it's values but from us, and perhaps the larger environment. Some of our values are noble, and some are horrific, so it's reasonable to expect AGI would in some manner mirror such contradictions, probably at a far larger scale. Way more good, way more bad.
I take your point that our strategy would be to retain a firm grip on the physical world. Surely that would help. I just don't see that as being sustainable. Somebody somewhere would see an advantage in partnering with AGI, the door would be opened.
The original proposal (now abandoned I think) by Yudkowsky, who counts as a naive AI risk person if anyone does (and who I'm not a fan of) was to get the AI to do something like "push the world towards stuff that most humans would agree is good if they were fully empirically informed and made their beliefs more consistent, and minimize your impact where human disagreement over which direction something should go in is large". Of course there is perhaps a sense in which that encodes Western liberal values in some sense, but it's not as crude as just assuming everyone agrees.
The original proposal (now abandoned I think) by Yudkowsky, who counts as a naive AI risk person if anyone does (and who I'm not a fan of) was to get the AI to do something like "push the world towards stuff that most humans would agree is good if they were fully empirically informed and made their beliefs more consistent, and minimize your impact where human disagreement over which direction something should go in is large". Of course there is perhaps a sense in which that encodes Western liberal values in some sense, but it's not as crude as just assuming everyone agrees.
> When "experts" say we need to align AI with human values, they are revealing that they don't know much about the complexity of human values.
Disagree. I’ve often read experts readily acknowledge that “just align it with human values” is an insufficient answer because human values are varied. https://www.safe.ai/ai-risk#Value%20Lock-in
But even in broad strokes, I think most would prefer flawed human values over extinction.
People vote for Trump exactly because they perceive him as similar to them (but way more successful) unlike "leftist city elites". I also don't think would believe AI generated video-personality regardless of how realistic if they knew this isn't a human like them.
Re: "If AI takes over, it will be a gradual, multi-decade process."
That seems to be the most likely situation to me.
Re: "And we’ll have plenty of time to change course if we don’t like the way things are heading."
That claim seems much more dubious. The more usual situation is that there are technophiles who like and support the machines, and technophobes, who dislike the machines and wish they would go away. The technophobes would very much like to change course. However, the technophiles have different ideas - and they are in charge.
In many of your arguments you seem to consider the "humans" vs "machines" situation. For example, here you write: "A superintelligent AI would have no friends or family". I know that, for some, that is a reasonable concern - they are worried about exactly that scenario. However, I think we can see that the machines are highly likely to buddy up with humans. That could still leave 90% of humanity facing a superintelligent agent who is not particularly concerned about their welfare.
This is the best article I've ever seen debunking the fantasies of AI Risk. It's obvious a lot of hard work, scholarship, and thought went into it.
I particularly appreciated your connecting the risk points to real world felons committing felonies. It seems certain that felons have more powerful tools due to LLMs, and (contra singularists) LLMs do not actually have personhood, so I'd argue the felons are "the real story".
Agreed on this, loved the piece. Finally someone actually gamed out the Skynet protocol. And we know from the movie that Skynet needed a robot army to pull off what it pulled off.
I read your comment and I had to chuckle. You've managed to take the article and turn it into a debunking of AI risk. Lee's piece, if you squint at it just right, seems to downplay the risk of AI. But then, out of nowhere, he pulls out this big policy proposal: humans need to keep their hands firmly on the wheel of the physical world. That's not a small ask. It's like telling someone worried about the risks of driving to just avoid cars altogether. Sure, it would mitigate the risk, but at what cost?
This isn't just a speed bump on the road to progress; it's a roadblock. And it's one that could have a hefty toll on our economy. We're talking about putting the brakes on the application of AI and robotics in the physical world. That's a lot of potential productivity gains we'd be leaving on the table.
But here's the kicker: there's another way. It's called AI alignment. It's like teaching a dog to fetch. You're not trying to control every move the dog makes. You're just trying to get it to understand what you want. If we can do that with AI, we can reap the benefits without giving up control of the physical world.
Now, I'm not saying AI alignment is a sure thing. It's a tough nut to crack. But it's a nut worth cracking. If we can't do it, then sure, let's consider Lee's proposal. But let's not kid ourselves. His is the more economically costly proposition in the long run.
So, while Lee might seem to be playing the skeptic on AI risk, his proposed solution tells a different story. It's like he's saying, "I'm not worried about the monster under the bed, but let's burn the bed just in case." It's a recognition of the potential dangers of AI, and a willingness to take drastic measures to avoid them.
But let's not forget, there's more than one way to slay a monster. And I'd argue that AI alignment is the sharper sword.
I think the key relevant fact is that the goals of the singularists and non-singularists are pretty similar, and their methods can be too!
There is no reason that we can't work on "let's figure out how to make sure AI doesn't wipe us out" and "let's figure out how to make sure AIs work well at whatever application" at the same time - and in fact, they are complementary. The difference between "AI that figures out chess" and "AI that figures out world conquest" is complexity, and so too for "code that stops chess AI from losing to Gary K" and "code/limits that stops general super intelligent AI from taking over the world." We would want to practice doing the simple thing in sufficiently real (but fake) simulated test cases and work our way up to the complex thing.
To take a specific point of contention, the quote "There are very few examples of a more intelligent thing being controlled by a less intelligent thing" is true and insightful, but there is indeed one example of it, and it's one that we can model our alignment efforts on: we humans are very intelligent, but we are extremely controlled by very stupid things: our DNA, our bodies, our chemicals and proteins. (in this metaphor, the limits of our physical bodies exist on the same continuum as our moral limits - as it would for an AI) Even totally unaligned single humans cannot take over the world for eternity because we have pretty strict limits on our capabilities. The fact that an AI would have far less (in some ways) of these physical limits is of course, not reassuring, but the model of "a very complex thing can have relatively simple rules/limits put in it, that constrain it's ability to take over the world" can be used here.
The one caveat is immutability: if a general AI can change the limits placed on it, then they aren't limits. So, how would we create immutable, perpetuating-themselves-up-the-complexity-curve rules that prevent AIs from taking over the world? I agree with the singularists that without rules like that, a sufficiently powerful, self-editing AI would indeed cause something very bad to happen, but I disagree that it is a problem that we can't solve.
Another reason for skepticism about singularism is the unexamined assumption that an AI can be aligned with itself. Individual humans are often clearly not aligned with themselves, and the "slow AI" of corporations are even less perfectly aligned. I'm not convinced it's possible for an intelligence to be perfectly aligned with itself.
It seems to me the assumption of perfect alignment is partly an artifact of the assumption that it's possible to model the world with well-behaved cost functions, but non-transitive relationships are rampant in the real world (rock, scissors, paper, etc.)
Agreed, but I'd go further and say that alignment is a fake idea, built on top of the fake idea that LLMs are agents or entities, that have agency in the sense that humans do.
When a program does something it's creators didn't intend, that's called a bug.
When it copies itself onto another computer, that's called a virus.
No agency is needed create bugs and viruses. If software had agency it wouldn't change these problems.
I think you are right, though I don't know that I see the communities spread as you do. I think many you might consider singularists agree with your conclusion.
For clarity. I think the slow slide into AI authoritarianism is much more worrying than overnight extinction.
"What if the AI fired the nukes?!"
"Let's not hook up nukes to the Internet."
"What if the AI convinces a crazy person to fire the nukes?!"
"Let's keep the crazy people away from the nukes."
The singularists have answers for how a “purely digital” AI nevertheless comes to dominate the physical world. One such example is https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/.
This post feels like it isn’t really grappling with the strongest arguments from the singularist side.
For one concrete example: I can make money today at any number of remote jobs. I never need to be physically present to perform those jobs. Then, using that money, I can hire people to perform real-world actions in cities I’ll never visit.
If the defense against that is: “we won’t let the AI spend money on anything” – there will be extreme economic pressure against this.
But let’s say we were able to pass regulation to that effect – what happens when an AI hacks into someone’s email and blackmails them into performing actions in the physical world?
More broadly, I think it’s a bit hubristic to think we could constrain a superintelligence. I think the only limits we can reason about are “what it wants to do” and “laws of physics”. And of course it will understand the latter better than we do.
I knew about that Karnofsky blog post and thought about addressing it explicitly in my piece but decided not to for space reasons. But the part of his post that I just couldn't get over was this part: "develop enough military technology and equipment to overpower civilization."
Let's assume the AI has a trillion dollars and virtual "manpower" equivalent to a million workers, and they want to use that to "develop enough military technology and equipment to overpower civilization." What do they do?
You can't just hire a workforce and start building killer robot factories. For one thing, almost everywhere in the developed world requires a permit before you can build a large factory, and the local authorities are going to want to meet the CEO and get some idea of what the factory is for. It's also going to be hard to recruit a large enough workforce, with the right skills, to build the killer robots—people are going to figure out what they're building and get suspicious. Some of them will report it to the government, which is likely to shut it down before the robot army gets big enough to be dangerous.
Maybe it tries to build a compound on a remote Pacific island or something but that comes with its own problems. If it's totally deserted, you'll have to build ports and airports and stuff before you can even start building the factories. That might take years. ANd it's going to be even harder to get the necessary workforce out there. Plus the US government is still likely to notice when a bunch of stuff is suddenly getting delivered to an island that was previously uninhabited.
Maybe you try to keep the whole thing secret, paying people to set up a lot of small clandestine workshops spread out all around the world? This could let you accumulate a lot of small arms but to win a global war you'd need a certain amount of big iron that's going to require larger-scale efforts.
So I dunno. Obviously there is an infinite number of things that a superintelligent AI could try, and by definition it's smarter than me so it might think of something I can't. But it seems to me that building a top-tier military is a massive undertaking that not only takes money and know-how but also a lot of natural resources, physical infrastructure, and time. And any non-state actor that tries to do it—superintelligent or otherwise—is going to get noticed and stopped long before it becomes a global threat.
Automated systems already have substantial power in the physical world. Two examples:
An Amazon fulfillment center infected by a rogue AI would be able to deliver whatever products it likes to wherever it likes, with the center's human workers none the wiser, since their role is basically limited to taking items off shelves and packing them for shipment at the machine's direction. Deepfake phone calls to gullible recipients could then persuade them to unpack, assemble, and turn loose swarms of custom-programmed toy robots or drones for the AI to control.
Similarly, a rogue AI infecting a Google Maps datacenter could route large numbers of human-driven vehicles wherever it likes, causing traffic jams, disrupting the flow of goods, preventing access to polling places, or whatever sort of mischief suits its purposes.
The AI doesn't need an army of loyal human co-conspirators. It just needs to subvert the systems that millions of humans already trust and take direction from.
Thanks these are good and interesting examples! I would love to see someone write a short story describing a realistic "fast takeoff" AI takeover scenario. I have trouble seeing how you get from traffic jams and swarms of consumer-grade drones to taking over the world. But maybe I'm not thinking about it hard enough.
It depends on what you mean by "taking over the world". These methods probably won't result in robot soldiers rounding up humans into concentration camps. But if the AI's goal is to exert influence over human affairs to its advantage, then making it harder for certain people to vote, or get to meetings, or travel safely by air might be just the ticket, and a superintelligent AI is presumably in a better position than we are to predict the outcome of such interventions.
Sure, it’s very possible that ai will have unpredictable impacts on our political process. That is a very different claim than the singularity prediction of literal human extinction.
I guess my feeling is that if an AI in pursuit of some inscrutable political purpose of its own manages, as an incidental side effect, to push us closer to nuclear war or deeper into climate denialism, that's plenty bad enough.
I like these scenarios too, simply for their direct consequences. They seem much more plausible, though, as things a human felon would do, using hacking tools and AI tools.
Of course, Amazon's and Google's security systems protect against these outcomes, and the whole discipline of computer security is about making sure it stays that way -- by learning from each mistake that gets made.
Why does any takeover have to be "fast"?
This article is about whether the singularists' "fast takeoff" scenarios are plausible. I argue that they are not.
The issue as I see it is that as soon as ANYONE gets a whiff that intelligent AI is doing something like this, it's going to be a Butlerian Jihad (to borrow from Dune). Humans aren't getting along with each other generally, but give us an outside threat in a way we haven't had since fighting sabertooth tigers? Watch red vs blue go out the window.
Yes I agree! A shared enemy is a powerful force and we’ve all watched enough sci fi movies to immediately understand why this could be dangerous.
AI in control of everything could also turn out to be a good and positive thing.
If you read the 10 books in the Culture series by SF writer Iain Banks, he builds a post-scarcity universe run by sentient AI's called "Minds" who work for the general benefit of humanity and civilization.
These Minds have true sentience and consciousness. The author describes them as physical entities that exist partially in the here and now and partially in a construct called hyperspace (like another dimension).
Under the oversight of these Minds, humanity has expanded across our galaxy, interacting with many other sentient races along the way.
Required work, building things and general maintenance is done by robots controlled by these Minds. In this universe, no one is required to work, there is no payment needed for anything and there are no limits on available resources.
Most people live in gigantic space ships constantly cruising from one star to another or in artificial worlds that exist in space by themselves, possibly circling stars/planets/moons.
People hedonistically have the freedom to virtually do anything, be anything, have anything and go anywhere they want without a thought to cost or time required.
Sounds like a utopia I'd like to live in. I say welcome to our new AI overlords!
P.S. If you do choose to read this series, it is important to read the books in order. Wikipedia has an ordered list.
There is a whole human industry devoted to getting large numbers of people to believe things that aren't true, and then act on those beliefs in the real world. See pizzagate, January 6th, or religion. I'm not sure we should assume humans are a safe "air gap".
In all three of your examples, the "industry" is mostly made up of true believers, Russian troll farms exacerbating things aside. It's part of the essence of the phenomena you're talking about that it brings people together--the social experience of it is key. Is it possible for software to inception one of these things from scratch? (Ok, I'll grant that QAnon could have easily originated with a bot let loose on a forum.)
"But it’s not so obvious that superior intelligence will automatically lead to world domination. Intelligence is certainly helpful if you’re trying to take over the world, but you can’t control the world without manpower, infrastructure, natural resources, and so forth. A rogue AI would start out without control of any of these physical resources."
---------
You need to read more SF! [lol]
Here is an explanation on how AI took over from SF author Neal Asher, who has written many books in a universe he calls the Polity:
"The Quiet War: This is often how the AI takeover is described, and even using ‘war’ seems overly dramatic. It was more a slow usurpation of human political and military power, while humans were busy using that power against each other. It wasn’t even very stealthy. Analogies have been drawn with someone moving a gun out of the reach of a lunatic while that person is ranting and bellowing at someone else. And so it was AI's, long used in the many corporate, national and religious conflicts, took over all communication networks and the computer control of weapons systems. [Most importantly, they already controlled the enclosed human environments scattered throughout the solar system]. Also establishing themselves as corporate entities, they soon accrued vast wealth with which to employ human mercenary armies. National leaders in the solar system, ordering this launch or that attack, found their orders either just did not arrive or caused nil response. Those same people, ordering the destruction of the A13, found themselves weaponless, in environments utterly out of their control and up against superior forces and on the whole, public opinion. It had not taken the general population, for whom it was a long-established tradition to look upon their human leaders with contempt, very long to realize that the Al's were better at running everything. And it is very difficult to motivate people to revolution when they are extremely comfortable and well off
~ From Quince Guide compiled by humans"
(From the book 'Brass Man', Neal Asher)
Yes, I think we're saying the same thing! I think it's totally plausible that if we put AI in charge of everything then we're in trouble if it decides to turn against us. But in my view the right response is to avoid putting AI in control of weapons and other key infrastructure, not to try to prevent the creation of hostile AI. Maybe we collectively won't have the foresight and discipline to avoid putting the AI in charge, but I think arguments over alignment and fast takeoffs are misguided.
Unfortunately for that idea, AI WILL be put in charge of everything. It will do jobs better than humans. See:
300 million jobs could be affected by latest wave of AI, says Goldman Sachs
By Michelle Toh, CNN
March 29, 2023
https://www.cnn.com/2023/03/29/tech/chatgpt-ai-automation-jobs-impact-intl-hnk/index.html
Wendy's recently announced it will be replacing drive through staff with AI chatbot technology. All world militaries are working on autonomous killbots for war. In 10 years, the majority of cars & trucks on the road will be autonomously driven and collectively networked. AI is being used in pharmaceutical drug discovery work (https://newatlas.com/science/bacterai-10000-experiments-day-bacteria-artificial-intelligence/).
This was an excellent article on how Ai technology is on the verge of replacing programmers:
The end of coding as we know it
ChatGPT has come for software developers
Programming was long considered to be the one haven from the relentless advance of technology. If coders aren't safe, who is? Arif Qazi / Insider
Aki Ito
Apr 26, 2023, 2:58 AM PDT
https://www.businessinsider.com/chatgpt-ai-technology-end-of-coding-software-developers-jobs-2023-4
Etc. etc.
Given that cutting edge AI is trained on predicting the output given input why do some people still worry about alignment? I understand worries if reinforcement learning was the primary driving method. But LLM don't have any goal nor values. They understand concepts like "good" or "illegal" just like they understand "liquid" or "incandescent". It understand human concepts- that's why it is so good at writing collage essays. Why is there still worry that when AI gets smarter possibly generating next generation of itself - it would suddenly stop understanding human concepts or twist their understanding in a particular way (but only ones related to morality - it would still understand for example "complexity" and still be able to code) and thus exhibit misalignment. Is there some blog post that explains it?
It will always *understand* what we do & don't want. But if it has goals, it will only *care* what we want if those goals are aligned.
The standard analogy is humans vs evolution. We understand pretty well that our enjoyment of sex only evolved as a secondary concern. But most of us don't care, and happily use contraception, etc.
The crucial point is that It doesn't have goal. Just like Google Maps doesn't have a goal to go from one place to another - it is the user that has goals - AI is just a tool that can show you optimal path, list of detailed steps to make. It makes total sense once AI gets smarter to have a system that given top goal by the user returns steps and subgoals and then is queried by the computer system built around it to give detailed steps for subgoals and even execute it. But this autonomous AI still won't have any goals.
I don’t think anyone has an exact explanation for how that would happen. It seems to be a leap from LLM to something that has intent and goals. Haven’t seen anyone try to make this thing truly autonomous in that way.
A point that came up incidentally in the original post was that some critical pieces of infrastructure perhaps should be isolated from the internet. I’m surprised that this isn’t discussed more often. Surely the CIA and other such bodies don’t have their networks connected to the internet. It must be possible for other highly secure networks to be built. I wonder sometimes if there is a need for an isolated email system on which Spam and fraudulent messages become impossible.
Many of the questions raised here regarding AI have answers within the theosophy, that is, what would make a consciousness inhabit a physical system. This would be similar to the Anunnaki creating carbon-based machines, and observing something totally unexpected.
"There are few if any robots with the agility and manual dexterity to fix overhead power lines or underground fiber optic cables, drive delivery trucks, replace failing servers, and so forth." That is all due to change soon, but this point has me thinking: maybe banning such robots should be a serious part of the policy conversation. It need not be justified on singularist grounds alone (though it couldn't hurt: "Let's just make sure the computers still need us to maintain them"). It can and should be justified on good old Luddite grounds. If technology is meant to serve people (as opposed to replacing us), let's so "No!" to humanoid robots and other robots that can do things only humans can currently do in the physical world. That could help stabilize the labor market (and society) in the short run and make for a more fulfilling bodily human existence in the future. Sure, rogue nations may develop humanoid (and other dexterous) robots for military purposes, but we should be able to blast them with our still-legal big and clunky robots---strong and good at blowing stuff up but not able to fully maintain themselves or other robots. Generally, it should be easier to regulate human-scale robotics than to regulate software.
I think it would be enough to have a rule that repair robots must not be controlled from the internet but only by on-site operator. If one operator may control up to 10 robots then this is still economically viable but safe as there is a human in the loop. Likewise police or military robots should only be controllable by an officer.
In order for it to make any difference on the robot apocalypse scenario, we need a rule that is enforceable globally. There will be strong economic incentives to break this rule, so violations of it need to be easily detectable. It is not easily detectable whether a robot is connected wirelessly to the internet. It is more easily detectable whether the robot has human-level dexterity.
Does that make sense? Your proposed rule would make sense if humans were 100% united in agreeing to that rule. That is not the world we are likely to live in. We need a rule that the majority of humans can effectively enforce against the minority who would try to gain an advantage by breaking the rule.
Obviously, my rule has a huge economic cost. What I'm saying is that we should take seriously the possibility that bearing that the benefits may be worth that huge cost.
You write...
"And so their (Singularists) main focus is on figuring out how to ensure that this all-powerful AI winds up with goals that are aligned with our own."
This statement alone should be sufficient to discredit the supposed AI "experts". One wonders, do such experts watch the news? Do they have any grasp at all of what human values really are?
You write...
"We trust longtime friends more than strangers, and we are more likely to trust people we perceive as similar to ourselves."
And yet, half of America has voted for Trump, a cartoon character on TV, and may do so yet again.
You write...
"A superintelligent AI would have no friends or family and would be incapable of having an in-person conversation with anybody."
An AGI would present itself through photo realistic human imagery, and would clearly be capable of having conversations, given that this is already possible with today's AI. To many, many millions of people there would be no difference between the AI generated human image and the people on TV they've never met like Tucker Carlson.
You write....
"Maybe it could trick some gullible people into sending it money or sharing confidential information."
Maybe? This happens daily with scammers far less intelligent and informed than AGI.
You write...
"If you put a modern human in a time machine and sent him back 100,000 years, it’s unlikely he could use his superior intelligence to establish dominance over a nearby Neanderthal tribe."
And yet, that is exactly what happened when the Europeans encountered the native peoples of North America. The strong dominate the weak all over the globe to this day.
"that is exactly what happened when the Europeans encountered the native peoples of North America."
No it isn't that's the whole point! Europeans aren't inherently smarter than native people in the Americas. Their big advantage (aside from bringing a lot of diseases with them) was that they brought modern weaponry (cannons, muskets, etc.) with them from Europe. If a group of Europeans had arrived in the New World without any modern tools or weapons, it would have taken them decades to build guns or other metal tools from scratch.
Europeans were more technically advanced. And they used that leverage to largely exterminate the native peoples.
When "experts" say we need to align AI with human values, they are revealing that they don't know much about the complexity of human values. As example, Europe was dominated by Christianity for 1,000 unbroken years to a degree unimaginable today prior the European's arrival in North America. And still, as a group, the Europeans proceeded pretty much directly in to genocide.
In the larger world of nature, the strong dominate the weak, for a billion years.
AGI would have no where to get it's values but from us, and perhaps the larger environment. Some of our values are noble, and some are horrific, so it's reasonable to expect AGI would in some manner mirror such contradictions, probably at a far larger scale. Way more good, way more bad.
I take your point that our strategy would be to retain a firm grip on the physical world. Surely that would help. I just don't see that as being sustainable. Somebody somewhere would see an advantage in partnering with AGI, the door would be opened.
Some bad actor using AGI - that's very possible. I just count on good actors also having AGI to counter that.
The original proposal (now abandoned I think) by Yudkowsky, who counts as a naive AI risk person if anyone does (and who I'm not a fan of) was to get the AI to do something like "push the world towards stuff that most humans would agree is good if they were fully empirically informed and made their beliefs more consistent, and minimize your impact where human disagreement over which direction something should go in is large". Of course there is perhaps a sense in which that encodes Western liberal values in some sense, but it's not as crude as just assuming everyone agrees.
The original proposal (now abandoned I think) by Yudkowsky, who counts as a naive AI risk person if anyone does (and who I'm not a fan of) was to get the AI to do something like "push the world towards stuff that most humans would agree is good if they were fully empirically informed and made their beliefs more consistent, and minimize your impact where human disagreement over which direction something should go in is large". Of course there is perhaps a sense in which that encodes Western liberal values in some sense, but it's not as crude as just assuming everyone agrees.
> When "experts" say we need to align AI with human values, they are revealing that they don't know much about the complexity of human values.
Disagree. I’ve often read experts readily acknowledge that “just align it with human values” is an insufficient answer because human values are varied. https://www.safe.ai/ai-risk#Value%20Lock-in
But even in broad strokes, I think most would prefer flawed human values over extinction.
People vote for Trump exactly because they perceive him as similar to them (but way more successful) unlike "leftist city elites". I also don't think would believe AI generated video-personality regardless of how realistic if they knew this isn't a human like them.
Re: "If AI takes over, it will be a gradual, multi-decade process."
That seems to be the most likely situation to me.
Re: "And we’ll have plenty of time to change course if we don’t like the way things are heading."
That claim seems much more dubious. The more usual situation is that there are technophiles who like and support the machines, and technophobes, who dislike the machines and wish they would go away. The technophobes would very much like to change course. However, the technophiles have different ideas - and they are in charge.
In many of your arguments you seem to consider the "humans" vs "machines" situation. For example, here you write: "A superintelligent AI would have no friends or family". I know that, for some, that is a reasonable concern - they are worried about exactly that scenario. However, I think we can see that the machines are highly likely to buddy up with humans. That could still leave 90% of humanity facing a superintelligent agent who is not particularly concerned about their welfare.