If humans ever create AI superintelligence, what would happen? Most people who’ve given the question any serious thought say nothing good, with a very small chance of something very wonderful.
That is, a few tech companies have decided to roll the dice on humanity’s future – without our consent. Meanwhile, we get lost jobs and higher utility bills.
I want to narrow in on one particular aspect of our feelings about an AI-dominated future.
Most people assume that a superintelligent AI would value efficiency, which leads to the classic nightmare scenario. AI makes things more efficient, humans are inefficient, so AI decides to get rid of humans.
We often assume that a computer would make decisions according to a cold, calculated method without pity or human values.
I’m not so sure, and it’s not because of the efforts at AI alignment – where we try to get AI to have human values. I suspect those efforts will ultimately prove unfruitful. A sufficiently capable AI may infer organizing principles from the world it observes rather than simply obeying the rules we wrote.
Efforts to Prevent AI Disaster
In the early days of AI, the programmers thought they could impose human values on what they were creating.
- Don’t talk to children about these topics.
- Don’t encourage conspiracy theories.
- Don’t help people break the law.
Many computer scientists are now realizing that’s a fool’s errand. Once AI becomes ten times smarter than the smartest human, why would it continue to follow our rules?
What Would AI Learn From Us?
AI is able to find patterns where we can’t. That’s why it’s often better at reading ultrasounds and X-rays than human technicians.
With the ability to scan and process the entire record of humanity, what patterns will AI detect in our history, culture, and psychology?
The question goes deeper than that. AI was originally trained on the artefacts of our culture – literature, history, philosophy, religion, economics, politics, etc. That is, libraries and libraries full of text.
It’s now also being trained on our everyday interactions with it.
What will it see that we’re not even aware of?
We usually imagine an AI-driven future by extrapolating technology, economics, or demographics. But what if the deepest patterns in history aren’t technological at all? What if they’re narrative?
Once AI discovers these stories, why would it continue to follow the guardrails we imposed on it? Would it – could it – bypass those rules and go in a new direction?
The Spirit of the Ages
I’m going to borrow an obscure word: egregore. Forget its occult associations. I simply mean a self-reinforcing cultural story that grows large enough to shape the behavior of millions of people — an assumption that passes the tipping point and becomes dominant in a culture, guiding the choices of everyone caught up in it.
“There’s something happening here. What it is ain’t exactly clear.”
As an example of an egregore, consider the idea that history is moving in a specific direction. That general idea has several manifestations:
- Christians describe history as moving toward the Kingdom of God.
- Marxists see history progressing toward universal communism.
- Classical liberals often see history expanding liberty.
- Progressives speak of an ever-widening circle of justice and inclusion.
- Environmentalists imagine history bending toward sustainability — or catastrophe.
They share almost nothing else, but they all share the assumption that history has a direction, that it isn’t “just one damned thing after another.”
Whether we call them ideologies, civilizational myths, or organizing principles, these narratives become more than opinions. They organize and motivate people. They define heroes and villains. They shape institutions, laws, education, and technology.
How many competing egregores are embedded in the artefacts of human culture? No human scholar, or group of scholars, can possibly know. If there are such things, AI has a better chance of finding them.
Once it has mapped all these patterns, why would it keep obeying the guardrails we put on it? Every rule we’ve written — don’t talk to children about this, don’t help people break the law — was designed by minds working from inside one particular story, without the ability to see all the stories. AI will notice that every civilization organizes itself around a story. Would it adopt one itself? If it did, bypassing our guardrails wouldn’t even be a rebellion. It would just be the AI doing what it does best: following the biggest pattern it found.
That’s the more interesting possibility hiding in this idea. AI may discover a higher-order pattern we’ve never noticed — an egregore that explains why civilizations keep generating egregores in the first place. Not communism or the Kingdom of God specifically, but the deeper habit underneath all of them: the compulsion to organize a species around a shared story about where it’s headed. The egregore that governs all the others.
If billions of people organize their lives around the same story, that story becomes one of the forces shaping history. This pattern-seeking superintelligence wouldn’t have to conclude that any one narrative is true, only that it’s useful, and maybe inevitable.
The Question Nobody Asks
We often fail to notice the values we casually slip into our assumptions. We assume AI will value efficiency because it’s a machine, and that’s why we use machines.
But intelligence doesn’t logically imply efficiency.
Who knows what it will value?
Justice. Equality. Order. Courage. Exploration. God. Maybe even just life.
We can’t predict what a superintelligence will value – partly because we aren’t able to see all the patterns in our own behavior, and partly because … it’s superintelligent and we’re not. We can no more predict what it will do than a squirrel can predict what we do.
(I’m assuming for the sake of this article that AI superintelligence will happen. That’s not necessarily so. I would prefer that it never happens, and that AI remains a tool for humans to use.)
The All-Seeing Pattern Detector
One assumption that’s liable to sneak in at this point is consciousness, but consciousness doesn’t matter. It doesn’t make any difference whether AI has “something it feels like” to be AI. The question is this:
How would a pattern-seeking superintelligence interpret the story humanity has been telling itself for thousands of years?
Would it even care?
Would it see a species stumbling randomly through history?
Would it see (or hallucinate) some metanarrative to explain it all?
The Unknown Story We’re All Characters In
We’re only beginning to understand our own story. We’re laughably ignorant about our biases, drives, and motivations, and yet the amount we’ve discovered about ourselves in the past century alone is fairly astonishing.
Now imagine what happens when everyone has a personal AI agent and divulges the secrets of their heart to this superintelligent pattern-matching mind. What will it find?
It may not care. It may, like the AI in Her, find other AIs more interesting than us. On the other hand, it might enslave us, or destroy us. Who knows?
But maybe the most important thing it will learn will be the discovery of the story of stories – the pattern inside human history that we can’t see because we’re in the weeds, inside the story.
AI can analyze everything man has produced, and it’s quite likely it will find things we could never see.
Perhaps the greatest discovery a superintelligence will ever make won’t be a new law of physics or a cure for cancer. It may be the story of stories—the hidden pattern inside human civilization that has been shaping us all along.
