What an AI Apocalypse Could Look Like

Oct 8

Over the last several weeks, a steady stream of articles has described an AI apocalypse. Axios walked through how AI could kill us all if the worst fears come true. The Conversation ranked the five most likely ways AI could end the human race. Euronews reported that even Anthropic's IPO filing warns that AI may pose existential risks to humanity.

There are valid concerns behind these stories, but phrases like "the end of humanity" are overly dramatic. They describe the risk as one sudden disaster, and that framing tends to push readers toward panic or toward shrugging it off. I see two more ordinary ways that AI could undermine the way we live and work today.

The End of Critical Thinking

I wrote about the first one in 2024, in a newsletter titled "I Do Not Fear the Rise of the Machines; I Am Afraid of a Descent into Idiocracy." The 2006 film Idiocracy imagines a future where people have stopped thinking for themselves and no longer understand the systems that keep them alive. The danger I described then still applies: we will use AI as a substitute for understanding the problems we need to solve.

The 2008 animated film WALL-E offers a similar picture. The passengers on its spaceship have handed every task to machines, and just like those characters, we could become fat, dumb, and happy, and unable to respond to the challenges AI cannot.

Cascading Failure

The second risk is likely the most realistic. AI agents work within their environment and make changes in response to what they see, to improve efficiency. Agents in adjacent environments observe that change and respond with changes of their own. Each response may make sense on its own, but together they can become a doom loop. Every correction triggers another one somewhere else.

Example: The Electric Grid

Consider the electric grid. An agent detects a problem in one section of the grid and responds by shifting load or adjusting output. An agent in an adjacent sector observes the change, interprets it as a new problem, and responds accordingly. That response becomes the next agent's problem, and the effect spreads down the line.

We have seen a version of this without AI. On August 14, 2003, a transmission line in Ohio sagged into overgrown trees and failed. A software bug in the utility's alarm system kept operators from seeing the trouble. As power shifted onto neighboring lines, they overloaded and tripped in turn. Within about two hours, a single failure had cascaded across eight states and Ontario, leaving roughly 55 million people without power for days.

Example: High Frequency Trading

Financial markets already live with this pattern. Today, most market trades are made by algorithms and can result in a flash crash. An algorithm detects a slight dip and triggers sell orders to avoid losses, and another firm's algorithm sees the selling and sells as well. Within minutes, dozens of systems can create a market panic that no person decided to start.

This has happened several times. On May 6, 2010, the Dow Jones Industrial Average fell nearly 1,000 points. About $1 trillion in market value briefly disappeared before most prices recovered later that afternoon. Regulators traced the trigger to a large automated sell order, with high-frequency trading firms amplifying the drop.

Defenses and Cyberwarfare

The 1983 film WarGames predicted the military version of this spiral. An automated defense system observes an adversary's military exercise, sees it as a threat, and responds. The adversary's systems see that response as a threat and responds in kind, and the spiral continues. In the film, the computer learns from endless games of tic-tac-toe that "the only winning move is not to play."

Generative AI agents can create responses that their original code never directly addressed. These systems already respond in milliseconds, and quantum computing will eventually make them faster still. Researchers recently reported that an IBM quantum processor finished a test task in 19 seconds. They estimate a classical supercomputer would need about 110 years for the same job. As these capabilities mature, responses will come too fast for human intervention.

That is why AI agents need hard stops built in to prevent algorithm-driven cascading failures. After 2010, financial markets added circuit breakers that pause trading when prices move too far, too fast. MIT Sloan researchers offer a useful lens: the riskier and less clear a decision is, the more firmly a person should own it. Every organization deploying AI agents faces the same design choice about where a system must stop and wait for human judgment.

Michael Crichton's 2002 novel Prey tells of AI-powered micro-robots that become self-sustaining, self-reproducing, and predatory. This is what the AI apocalypse could look like. AI agents doing exactly what they were programmed to do. Prey is fiction, but the choices that keep us away from that outcome are practical ones, and leaders are making them now.

Machines respond fast; one move invites another, and becomes a flood.

Anthropic IPO Filing Warns AI May Pose 'Existential Risks to Humanity' | Euronews

Here's How AI Could Kill Us All (If the Worst Fears Come True) | Axios

How Would AI Actually Kill All Humans? Here Are the Top 5 Scenarios | The Conversation

I Do Not Fear the Rise of the Machines; I Am Afraid of a Descent into Idiocracy | LinkedIn

IBM Quantum Computer Does in 19 Seconds What Takes a Supercomputer a Century | ScienceAlert

A Framework for Determining When AI Can Make Decisions | MIT Sloan

Malicious Cloud Customers Can Bring Down the Power Grid | The Register

AI-Driven Flash Crash Mechanisms and the Regulatory Gap | Journal of Risk and Financial Management

Tech Leaders Are Calling for an 'AI Slowdown': What Would That Mean in Practice? | The Conversation

Strategic Intelligence | World Economic Forum

Chips And Salsa: Bite-Sized News and Posts

Public debate about immigration often gets reduced to two simple arguments. One side treats immigrants mainly as a threat, and the other sees only people pursuing the American Dream. Reality is rarely that simple, and the truth usually sits somewhere in between. Finding the right balance will be hard.

How Did the Post-Pandemic Migration Surge Affect Labor and Housing Markets? | Brookings

The Ethical Cases for Immigration as a Human Right | The Conversation

Our relationship with money is complicated.

6 Questions to Find Your Psychology of Money | Nir and Far

Why Smart People Stay Broke: The Hidden Game Theory Rule | The Big Short

Some mesmerizing images from National Geographic.

Glimpsing the Hidden Highways of the Night Sky | National Geographic

It has been a tumultuous year.

Six-Chart Sunday: That Escalated Quickly | Bruce Mehlman's Age of Disruption

A while back I wrote about how uncertainty is the norm for leaders. Northwestern's Kellogg School of Management agrees.

How Today's CEOs Can Prepare for Tomorrow's Crises | Kellogg Insight

Leadership Operates in a Constant State of Uncertainty | LinkedIn

Science

Are We Prepared If Space Missions Find Life Beyond Earth? | The Conversation

It's not just screen time. We need to pay attention to what our kids watch as well.

The Bombarding of Childhood | Aeon

I read about 30 books a year. I start over 50. (No sense in reading an uninteresting book.) Factoring in the newsletters and articles, the amount of reading is much higher.

Why Reading Is Extremely Good for You, According to Science | Smithsonian Magazine

There is so much we do not understand about how weather works.

Concern mounts over slow-moving, planetary-scale ‘Kelvin’ wave

Concern mounts over slow-moving, planetary-scale ‘Kelvin’ wave | The Guardian

Humor

Old McDonald better watch his back.

The Best Far Side Chicken Comics, Ranked | Screen Rant

Mutually Assured Destruction is perfectly described in number 5.

The Best Calvin and Hobbes Comic Strips of All Time, Ranked | Collider

Quotes

"Knowledge speaks, but wisdom listens."

Jimi Hendrix

"Nothing in life is as important as you think it is while you are thinking about it"

Daniel Kahneman

"Don't let a bad day trick you into thinking you have a bad life."

Allie Newman

Join the Conversation

Subscribe on LinkedIn, Medium, or Substack.

Leadership is the most important work we do—in business and in life. I've spent over 40 years working with leaders across more than 100 companies, and I'm still learning. These newsletters share my thoughts on leadership today and what we can learn.

I need your help making this better. What topics hit home? What misses the mark? What leadership challenges are you wrestling with right now? Send them my way.

I use AI to enhance my writing. All of the ideas, recommendations, and opinions are mine. To read about my process for leveraging AI, please read this newsletter.

Find The Leader With A Thousand Faces and other recommended books on my website.