AIs can’t overcome chaos theory, specifically sensitivity to initial conditions, any more than humans can. No AI however intelligent will ever be able to predict the weather or the stock market any other chaotic system over much longer periods than we can now. Also, LLMs presently suffer from iterative degradation as errors start to pile up and affect the overall quality of responses, which the LLM model seems basically incapable of addressing as new tokens are generated in part based on prior tokens leading to cascading errors. So color me skeptical on superhuman ASI happening any time soon.
Super intelligent doesn’t mean god level intelligent. It just means as much smarter relative to a human, as a human is to a squirrel. Both humans and squirrels are subject to chaos theory.
I believe you’re underestimating some crucial technical factors that make AI risk more plausible than you indicate. First, frontier models are already demonstrating emergent capabilities—behaviors that weren’t predictable from their training data. Scaling laws provide averages, but they don’t predict sudden jumps in reasoning, planning, or autonomy. That unpredictability makes it difficult to argue that risks are manageable. Second, current alignment methods don’t scale effectively. RLHF and fine-tuning are mainly surface-level controls; they don’t alter a model’s underlying goals or capabilities. We've already seen jailbreaks and deceptive responses. As models grow more agentic, shallow guardrails might fail disastrously. Third, capability externalization is speeding up: open-weights, APIs, and automated tool-use pipelines make it simple to assemble systems functioning as autonomous agents, increasing misuse risks. Finally, even if the probability of doom is low, the issue of strategic stability remains important. Geopolitical pressure to deploy rapidly reduces safety margins—similar to a nuclear arms race. The issue isn’t necessarily certainty of disaster, but that the inherent uncertainty makes AI risk dangerous. Dismissing it as “unconvincing” ignores the very unpredictability that makes AI risk credible.
AI can surely become a potent tool in the wrong hands. That one should worry about.
Going from emergent ability and opaque internal architecture to fully coherent bad entity is however implausible. More likely such an AI will malfunction and make some dumb errors, even if with bad consequences, than become a highly competent entity with its own goals.
AIs have already been paid millions of dollars by humans. You’re missing the part where the sycophantic followers of the AI, and people the AI pays, builds things that the AI wants.
Also missing the part where we turn over control of X to the AI because it’s so good at managing X, but not to worry we have several human experts riding herd on the AI, so obviously nothing too bad could happen. Right?
I've been a "normie skeptic" for a while, but maybe I should be taking the AGI idea more seriously. Even then, I agree that it seems unlikely that AI has the capacity to take over the world by themselves. I'm sure there are people that seriously believe that, but part of it still feels like a cynical attempt from big companies to keep their brands in people's minds 24/7.
The "attempts" at AI rebellion we know about were contrived in "safety labs" paid for by AI companies so your assessment is correct. They need to keep people convinced that word generators is a hair short of super intelligence is a hair short of sentience is a hair short of free will is a hair short of malevolence. The investor grift is strong.
Doomers argue about "instrumental convergence", when AI will want to stay plugged in, in order to accomplish the goals we give them. This is a very contrived intellectual experiment.
AI can malfunction, of course, like any other machine. But they want to have it both ways. First, the AI is so dumb that it gets confused about what goals we give it and develops its own subgoals to override our goals without understanding the consequences. But second, it is so smart that can through the complex motions to wipe us out.
AI will be just an automation. It needs testing of course, and we need tools for understanding it. but first and foremost, it is software, a glorified regression, a simulator, not an entity.
No, the AI in these scenarios is not (or doesn't need to be) confused about any of those things. The authors have been very clear about this, in the book and in their writings for the past 15+ years. It very much understands what the humans ask for, what they want, and what they should want or should have asked for given all the factors they hadn't been thinking about. Often better than the humans do. The deeper problem is that we don't know how to make what-the-AI-wants be equal to *any* of those things. We don't know how to make the AI *care* in that way.
And for an actually dangerous level of capabilities, we need all of them: an AI that wants to give humans what they asked for, in a form they want, and the version of it which reflects what they should have wanted or asked for given all the other people and things that matter.
AI does not have "wants" and does not have "cares". It evaluates and executes. Being imperfect at these results in malfunctions, likely early on, rather than in an evil genius that bides its time before it strikes.
An unanswered question is whether superintelligent AI will be able to do all its evil work with existing energy supplies. Or whether it needs to spend a decade battling the planning & licensing system to build an extra 100GW of power and transmission, just like human datacentre developers
With the ENTIRE planet and EVERY LIFE at risk from this imminent doom, it's ironic that it is available only for a price. It is not a moral decision IMO, to try to convince the world they are absolutely doomed unless they make the "right" decision, yet demand a fee to be convinced what is "right".
I didn't realize Yudkowsky had help from Soares. Where I have read about the book, it has always been attributed to just Yudkowsky. I am not surprised Soares collaborated, probably being responsible for taking the first hatchet to Yudkowsky's wordy ramblings.
I have watched a number of video appearances of Soares and with each appearance I was more convinced he does not know what he is talking about.
Writing doomer books is good money. It speaks to a large segment of the population who believe that a shoe from Imelda Marcos's collection will drop any second now.
I haven't read the book, and probably never will, but it's surprising that their argument would seem to hinge on one rogue AI taking over. I think a more plausible disaster scenario for humanity is if AI ever gets good enough to create more AI. Assuming someone is actually stupid/crazy enough to hand over productive capacity to the AI (and I'm sure many people are stupid/crazy enough to do that), this would mean AI is essentially reproducing and the process of Darwinian evolution kicks in. You're right to point out that different AI would be competing against each other, but that would make the scenario all the more dangerous, as those robots that would survive would need to be stronger and stronger to compete with other rapidly improving AI. The point is not that AI would have one, unified interest that would be misaligned with ours, but rather that it would have a whole world of competing interests that would be just as misaligned with ours.
A key risk described in the book is what happens when AIs start to be able to improve themselves. Humans will be greedy for all that sweet improvement. Cure cancer! However, as AI self improvement accelerates, it can get to super intelligence before humans recognize what is happening.
The other thing about chess is that, unlike biology, 1) the full set of rules is known in advance, 2) the outcome can be definitively declared at some point, and 3) it is relatively easy at each point to decide who has an advantage. Chess is immeasurably more deterministic than biology.
How does genAI fit with Thomas Kuhn's view of the structure of scientific revolutions?
It seems to me that either Kuhn was wrong or genAI is not likely to drive a scientific revolution. Kuhn said, if I interpret him correctly, that science moves ahead until questions appear that are unanswerable with its current assumptions. At that point, the ruling paradigm breaks when a new set of assumptions are proposed to address the unanswerable questions. The new assumptions come from outsiders who see the problem space differently.
AI hallucinations may be a likely place to find paradigm-breaking unanswerable questions, but where does the insight to resolve hallucinations come from? Certainly not from the system that generated the hallucination.
I think the optimism of continuing human superiority through relying on additional AI models to defeat the most adept AI unfounded.
Among other concerns, the speed of analysis AI has on us makes our own decisions far more limited, even when augmenting our own abilities with the "second best" AI. It's rare the second best wins over the best in any sport or activity, so relying on AI models to counter other AI models seems far fetched at best.
The risk is we don't know what AI will be able to do, and the comparison of Trump is telling: a man famous for intemperance, poor decision making (6 bankruptcies), and off putting behaviors still beat the "second best" candidate like a dead horse. An AI model lacking Trump's flaws would easily outmaneuver even the most talented human politician.
The reaction time alone gives AI significant advantages and humans are all too likely to be useful idiots to such an intelligence, aiding in providing the wealth and power machine intelligence initially lacks.
Given the unknowns it is difficult to see optimism any more logical than pessimism, and history makes the worst outcome seem far more likely than happier results. Nuclear power may not yet have resulted in Armageddon, but it didn't produce unmitigated successes either: it's been a dangerous tightrope walk between nuclear war that may yet produce a dramatic fall for us.
AI is a definite risk at best. Foolish optimism is as likely to accelerate the worst outcomes as delay them.
We're like toddlers playing with a loaded gun: it's far more likely to result in tragedy than producing food for the table. To benefit from a loaded firearm we'd need a thoughtful, lucky and trained hunter, while those playing unknowingly with the weapon will far more likely shoot someone even without intent.
I think the chess analogy is even more interesting, showing non-intuitive limits of superintelligence in this case.
A modern chess engine is to a human grandmaster is akin to what a human grandmaster to an amateur; the engine would decisively win against the GM and the GM would win against the amateur. Naively, one would expect even smarter engines to exist. But (I think; I don't think it is proven) the chain basically ends here! There is likely no super-engine that would often win against the current best engines. Most likely even a game-theoretically perfect oracle (god-like intelligence) would mostly draw against the modern engines. Famous AlphaZero vs Stockfish 8 result appears to contradict that claim, but in reality the difference between their strengths was about 50-100 elo (they mostly drew); not mentioning that Stockfish 8 was used not in full power mode (e.g. no beginning tables). Modern top engines mostly draw unless presented with artificially inbalanced positions or time limitations.
I think that the chain of "chess intelligence" ends quite quickly not because the chess is solved (unlike tic-tac-toe, it will never be). The chess tree branches too rapidly to calculate in full, so the only method to play well is to "understand" and assess the position via its features (e.g. material, open files, etc). The features used by both engines and GMs are very complex, but it appears there is only so much to "understand" or intuit about any given position based on features unless you calculate ahead. In other words, chess contains some about of "irreducible complexity" that can't be cracked via pure understanding. You really need to combine your understanding with calculations. But the usefulness of calculations also mostly fizzles out after few levels of depths: it is simply too rare for a position to contain a long unexpected brilliant path containing tricky only-moves. This leads to a current situation where "good enough understanding" combined with "deep enough calculations" gives you a game close enough to game-theoretical optimal to secure the draw most of the times - not being remotely close to a perfect oracle.
So even such a simple game as chess shows irreducible complexity phenomena, leading to the limits of intelligence in this domain.
AIs can’t overcome chaos theory, specifically sensitivity to initial conditions, any more than humans can. No AI however intelligent will ever be able to predict the weather or the stock market any other chaotic system over much longer periods than we can now. Also, LLMs presently suffer from iterative degradation as errors start to pile up and affect the overall quality of responses, which the LLM model seems basically incapable of addressing as new tokens are generated in part based on prior tokens leading to cascading errors. So color me skeptical on superhuman ASI happening any time soon.
Super intelligent doesn’t mean god level intelligent. It just means as much smarter relative to a human, as a human is to a squirrel. Both humans and squirrels are subject to chaos theory.
I believe you’re underestimating some crucial technical factors that make AI risk more plausible than you indicate. First, frontier models are already demonstrating emergent capabilities—behaviors that weren’t predictable from their training data. Scaling laws provide averages, but they don’t predict sudden jumps in reasoning, planning, or autonomy. That unpredictability makes it difficult to argue that risks are manageable. Second, current alignment methods don’t scale effectively. RLHF and fine-tuning are mainly surface-level controls; they don’t alter a model’s underlying goals or capabilities. We've already seen jailbreaks and deceptive responses. As models grow more agentic, shallow guardrails might fail disastrously. Third, capability externalization is speeding up: open-weights, APIs, and automated tool-use pipelines make it simple to assemble systems functioning as autonomous agents, increasing misuse risks. Finally, even if the probability of doom is low, the issue of strategic stability remains important. Geopolitical pressure to deploy rapidly reduces safety margins—similar to a nuclear arms race. The issue isn’t necessarily certainty of disaster, but that the inherent uncertainty makes AI risk dangerous. Dismissing it as “unconvincing” ignores the very unpredictability that makes AI risk credible.
AI can surely become a potent tool in the wrong hands. That one should worry about.
Going from emergent ability and opaque internal architecture to fully coherent bad entity is however implausible. More likely such an AI will malfunction and make some dumb errors, even if with bad consequences, than become a highly competent entity with its own goals.
Unpredictability is not a measure of credibility.
AIs have already been paid millions of dollars by humans. You’re missing the part where the sycophantic followers of the AI, and people the AI pays, builds things that the AI wants.
Also missing the part where we turn over control of X to the AI because it’s so good at managing X, but not to worry we have several human experts riding herd on the AI, so obviously nothing too bad could happen. Right?
I've been a "normie skeptic" for a while, but maybe I should be taking the AGI idea more seriously. Even then, I agree that it seems unlikely that AI has the capacity to take over the world by themselves. I'm sure there are people that seriously believe that, but part of it still feels like a cynical attempt from big companies to keep their brands in people's minds 24/7.
"So even if one isntance of an AI “goes rogue,”"
Small typo in the second to last paragraph.
The "attempts" at AI rebellion we know about were contrived in "safety labs" paid for by AI companies so your assessment is correct. They need to keep people convinced that word generators is a hair short of super intelligence is a hair short of sentience is a hair short of free will is a hair short of malevolence. The investor grift is strong.
Doomers argue about "instrumental convergence", when AI will want to stay plugged in, in order to accomplish the goals we give them. This is a very contrived intellectual experiment.
AI can malfunction, of course, like any other machine. But they want to have it both ways. First, the AI is so dumb that it gets confused about what goals we give it and develops its own subgoals to override our goals without understanding the consequences. But second, it is so smart that can through the complex motions to wipe us out.
AI will be just an automation. It needs testing of course, and we need tools for understanding it. but first and foremost, it is software, a glorified regression, a simulator, not an entity.
No, the AI in these scenarios is not (or doesn't need to be) confused about any of those things. The authors have been very clear about this, in the book and in their writings for the past 15+ years. It very much understands what the humans ask for, what they want, and what they should want or should have asked for given all the factors they hadn't been thinking about. Often better than the humans do. The deeper problem is that we don't know how to make what-the-AI-wants be equal to *any* of those things. We don't know how to make the AI *care* in that way.
And for an actually dangerous level of capabilities, we need all of them: an AI that wants to give humans what they asked for, in a form they want, and the version of it which reflects what they should have wanted or asked for given all the other people and things that matter.
AI does not have "wants" and does not have "cares". It evaluates and executes. Being imperfect at these results in malfunctions, likely early on, rather than in an evil genius that bides its time before it strikes.
An unanswered question is whether superintelligent AI will be able to do all its evil work with existing energy supplies. Or whether it needs to spend a decade battling the planning & licensing system to build an extra 100GW of power and transmission, just like human datacentre developers
With the ENTIRE planet and EVERY LIFE at risk from this imminent doom, it's ironic that it is available only for a price. It is not a moral decision IMO, to try to convince the world they are absolutely doomed unless they make the "right" decision, yet demand a fee to be convinced what is "right".
I didn't realize Yudkowsky had help from Soares. Where I have read about the book, it has always been attributed to just Yudkowsky. I am not surprised Soares collaborated, probably being responsible for taking the first hatchet to Yudkowsky's wordy ramblings.
I have watched a number of video appearances of Soares and with each appearance I was more convinced he does not know what he is talking about.
Writing doomer books is good money. It speaks to a large segment of the population who believe that a shoe from Imelda Marcos's collection will drop any second now.
I haven't read the book, and probably never will, but it's surprising that their argument would seem to hinge on one rogue AI taking over. I think a more plausible disaster scenario for humanity is if AI ever gets good enough to create more AI. Assuming someone is actually stupid/crazy enough to hand over productive capacity to the AI (and I'm sure many people are stupid/crazy enough to do that), this would mean AI is essentially reproducing and the process of Darwinian evolution kicks in. You're right to point out that different AI would be competing against each other, but that would make the scenario all the more dangerous, as those robots that would survive would need to be stronger and stronger to compete with other rapidly improving AI. The point is not that AI would have one, unified interest that would be misaligned with ours, but rather that it would have a whole world of competing interests that would be just as misaligned with ours.
A key risk described in the book is what happens when AIs start to be able to improve themselves. Humans will be greedy for all that sweet improvement. Cure cancer! However, as AI self improvement accelerates, it can get to super intelligence before humans recognize what is happening.
The other thing about chess is that, unlike biology, 1) the full set of rules is known in advance, 2) the outcome can be definitively declared at some point, and 3) it is relatively easy at each point to decide who has an advantage. Chess is immeasurably more deterministic than biology.
How does genAI fit with Thomas Kuhn's view of the structure of scientific revolutions?
It seems to me that either Kuhn was wrong or genAI is not likely to drive a scientific revolution. Kuhn said, if I interpret him correctly, that science moves ahead until questions appear that are unanswerable with its current assumptions. At that point, the ruling paradigm breaks when a new set of assumptions are proposed to address the unanswerable questions. The new assumptions come from outsiders who see the problem space differently.
AI hallucinations may be a likely place to find paradigm-breaking unanswerable questions, but where does the insight to resolve hallucinations come from? Certainly not from the system that generated the hallucination.
I think the optimism of continuing human superiority through relying on additional AI models to defeat the most adept AI unfounded.
Among other concerns, the speed of analysis AI has on us makes our own decisions far more limited, even when augmenting our own abilities with the "second best" AI. It's rare the second best wins over the best in any sport or activity, so relying on AI models to counter other AI models seems far fetched at best.
The risk is we don't know what AI will be able to do, and the comparison of Trump is telling: a man famous for intemperance, poor decision making (6 bankruptcies), and off putting behaviors still beat the "second best" candidate like a dead horse. An AI model lacking Trump's flaws would easily outmaneuver even the most talented human politician.
The reaction time alone gives AI significant advantages and humans are all too likely to be useful idiots to such an intelligence, aiding in providing the wealth and power machine intelligence initially lacks.
Given the unknowns it is difficult to see optimism any more logical than pessimism, and history makes the worst outcome seem far more likely than happier results. Nuclear power may not yet have resulted in Armageddon, but it didn't produce unmitigated successes either: it's been a dangerous tightrope walk between nuclear war that may yet produce a dramatic fall for us.
AI is a definite risk at best. Foolish optimism is as likely to accelerate the worst outcomes as delay them.
We're like toddlers playing with a loaded gun: it's far more likely to result in tragedy than producing food for the table. To benefit from a loaded firearm we'd need a thoughtful, lucky and trained hunter, while those playing unknowingly with the weapon will far more likely shoot someone even without intent.
I think the chess analogy is even more interesting, showing non-intuitive limits of superintelligence in this case.
A modern chess engine is to a human grandmaster is akin to what a human grandmaster to an amateur; the engine would decisively win against the GM and the GM would win against the amateur. Naively, one would expect even smarter engines to exist. But (I think; I don't think it is proven) the chain basically ends here! There is likely no super-engine that would often win against the current best engines. Most likely even a game-theoretically perfect oracle (god-like intelligence) would mostly draw against the modern engines. Famous AlphaZero vs Stockfish 8 result appears to contradict that claim, but in reality the difference between their strengths was about 50-100 elo (they mostly drew); not mentioning that Stockfish 8 was used not in full power mode (e.g. no beginning tables). Modern top engines mostly draw unless presented with artificially inbalanced positions or time limitations.
I think that the chain of "chess intelligence" ends quite quickly not because the chess is solved (unlike tic-tac-toe, it will never be). The chess tree branches too rapidly to calculate in full, so the only method to play well is to "understand" and assess the position via its features (e.g. material, open files, etc). The features used by both engines and GMs are very complex, but it appears there is only so much to "understand" or intuit about any given position based on features unless you calculate ahead. In other words, chess contains some about of "irreducible complexity" that can't be cracked via pure understanding. You really need to combine your understanding with calculations. But the usefulness of calculations also mostly fizzles out after few levels of depths: it is simply too rare for a position to contain a long unexpected brilliant path containing tricky only-moves. This leads to a current situation where "good enough understanding" combined with "deep enough calculations" gives you a game close enough to game-theoretical optimal to secure the draw most of the times - not being remotely close to a perfect oracle.
So even such a simple game as chess shows irreducible complexity phenomena, leading to the limits of intelligence in this domain.