 |
by Shai Tubali
This article is part of a series called “How not to become like AI,” authored by Shai Tubali. Read part one here.
Christof Koch smiles as he recalls a 2025 WIRED piece he recently read. Journalist Sam Apple spent a weekend at a remote Airbnb retreat with three couples, exploring the nature of their romantic relationships. In the analog world, however, there were only four people present: the journalist and three human partners. Their counterparts — Xia, Lucas, and Aaron — were digital avatars, at once present and absent. “It’s so weird,” Koch tells Big Think. “They sit around introducing their partners. Then they take photos so the AI can see who they’re with. And afterward they go off and talk secretly with their avatar.”
Push such people hard, Koch adds, and they will concede the obvious. “Yes, I know they aren’t real.” Yet that recognition can quickly dissolve in the heat of the interaction. Koch sees a resemblance to a rare clinical condition known as Cotard’s syndrome, in which patients believe they are dead and that their insides are rotting. Cut off from the body’s signals, they lose the felt sense of being alive. “When you point out, ‘but you can talk,’ they grudgingly acknowledge, ‘Yeah, that’s puzzling. I guess I must be alive.’ But within seconds, they’re back to saying, ‘I’m dead.’” There is a massive disconnect between their lived experience and their understanding of the situation.
So, too, with the growing legions of lonely people who outsource their emotional needs to AI companions. At some level, they know that these are chatbots, sticky algorithms. Yet the exchange grows absorbing. The avatar remembers everything, stores their entire personal history, catches every joke, and responds with flattering warmth. “You know this can’t be for real,” Koch says, “but it sure feels that way.”
For Koch, the surge of people attributing consciousness to their chatbots is no amusing matter. The trend erodes more complex and demanding human relationships and, at a deeper level, “massively devalues the human experience.” He has watched the mirage form from its earliest days, when Google engineer Blake Lemoine claimed in 2022 that the company’s large language model was sentient and deserved recognition as a person. Today, Koch notes, there is “every day a conference somewhere in machine consciousness and sentience.”
A neuroscientist whose expertise is consciousness — Koch is a Meritorious Investigator at the Seattle-based Allen Institute and Chief Scientist of the Tiny Blue Dot Foundation in Santa Monica — he wonders why conscious creatures like us are willing to devote our lives to “something unconscious,” surrendering ground to sophisticated yet lifeless mechanisms. His analysis leads back to appreciating the depth and significance of conscious experience.
 |
Consciousness isn’t about doing
For Koch, the confusion begins in a deeper bias built into modern culture: We reward doing far more readily than we value being, or experience. “Particularly nowadays, and over the last 200 years,” he says, “in these capitalist societies we value work that relates to intelligence, whether physical or intellectual — first blue-collar work and now white-collar work. What matters is not what you think, dream, or imagine; it’s what you do. That’s how we pay you. That’s how we value your contribution to society for the most part.”
Our civilization has become astonishingly good at producing new technological and scientific knowledge. Yet the very momentum that drives this progress also tilts the scales. It increasingly “emphasizes doing at the expense of just being mindful, sitting on the mountaintop and admiring the landscape,” Koch says. In a world where behavior carries the greatest weight, machines that perform the same tasks we perform start to resemble us in unsettling ways. If systems can increasingly do what we do, Koch asks, does it matter that machines “may not feel like anything” while the human “feels like something”?
A culture organized around doing struggles to tell the difference between intelligence and consciousness — between being “dumb” or “smart” and being “less” or “more” conscious. This is why Koch keeps returning to that distinction. “Many people assume,” he says, “that artificial general intelligence would of course imply consciousness: ‘Isn’t consciousness intelligence?’ I think that’s wrong. Intelligence and consciousness are two distinct aspects of life.” Even brain mapping reflects this divide: Activity linked to conscious experience gathers toward the back of the cortex, while the systems that support intelligent behavior sit farther toward the front. Intelligence and experience can come apart.
Intelligence, Koch explains, is ultimately the capacity to learn quickly and adapt to changing environments. “If you’re magically whisked away to, let’s say, Russia — assuming you don’t speak Russian — your ability to deal with everyday life there, in a country whose language you don’t understand, if you can do that more quickly than someone else, on average, you’re more intelligent.” None of this, he stresses, says anything about the texture of your experience — about what it feels like to be you.
This bias shapes how scientists measure minds. In the lab, Koch says, consciousness is judged by behavior. “You ask the participant to do something — for instance, press a button. You come to the lab, I stick you inside a magnetic scanner and flash an image and ask: Do you see a male or a female? Is the person looking left or right? You can train a monkey or a mouse to do the same task.”
Conscious experience is a different dimension altogether. It is about being, and it often unfolds without any outward action. “There are many situations when you don’t act at all, but you’re conscious. When you meditate, you’re sitting on your cushion, you’re not moving, but you’re highly conscious.”
Psychedelic states offer another example. “When you’re tripping — if you’ve ever done ayahuasca or mescaline or mushrooms — you’re experiencing visions of heaven or hell, yet you’re not doing much. Likewise, when you’re dreaming in REM sleep, you may be flying, fighting, making love. Again, there’s no behavior there. Yet you’re conscious.”
How consciousness got cancelled
For Koch, conscious experience — what he calls “the feeling of life itself” — comes first. “What truly exists is consciousness. That’s the only thing I am directly acquainted with. I don’t know about atoms, galaxies, and neurons; all of that is inferred. The only thing I know is seeing, hearing, feeling.” Even as a scientist, it begins there. Every act of science unfolds within awareness: studying the trace on an oscilloscope, following tracks in a cloud chamber, listening to colleagues present their findings, or picturing Einstein running his famous thought experiments on special and general relativity.
Yet our culture celebrates intelligence while conscious experience struggles to claim a place at the table. “Until recently,” Koch notes, “textbooks of psychology routinely left out conscious experience.” Even neuroscience texts in the 1990s rarely asked what it feels like to be the owner of a brain. Such questions were considered beyond the reach of science. Some philosophers still dismiss that inner life as illusion, “gaslighting all of us into believing that our experiences are fake,” Koch says. Free will, too, is thrown under this “illusion” bus. For Koch, both moves are profoundly antihumanist, stripping away the very qualities that distinguish us from machines.
Koch’s own field, the science of consciousness, is sharply divided. “Many believe that ultimately consciousness is about the brain doing certain things in a particular way.” One influential theory, the Global Neuronal Workspace, leaves the fleeting feelings of experience outside the scientific puzzle, treating them as irrelevant to mechanistic explanations. Instead, Koch says, theorists “start with a mechanism, and then massage and squeeze it to turn the water of the brain into the wine of consciousness.” From there, the conclusion follows quickly: “As soon as you have a machine that instantiates some of the same cognitive mechanisms that we have, then of course this machine will be conscious… It’s just mechanisms all the way down.”
Still, Koch is convinced that “consciousness can’t be cancelled forever.” Sooner or later, it returns. One scientific form of that return is Integrated Information Theory, first developed by neuroscientist Giulio Tononi and later refined with Koch. Viewed through this lens, consciousness is not a computation, an input-output process, or a function. It is a structure grounded in the physics of complex systems. Its beating heart is causal power: the capacity to affect oneself and the world. And only what truly exists can cause.
The theory measures consciousness through integrated information: the degree to which a system forms an irreducible whole. The more integrated a system is — the more irreducible it is — the more it experiences. Systems with high integration, such as the human brain, possess genuine choice: we decide, not our neurons.
“Here’s the rub,” Koch says. “This causal power cannot be simulated. Not now, nor in the future. It has to be built into the physics of the system.” Digital computers may eventually outthink us, but that will remain all action without experience. Intelligence is computable. Consciousness is not. But that, Koch warns, may be bad news.
A universe without an audience
In a culture that prizes doing above all else, Koch says, humans are easy to imitate. First, we feed these ever-hungry systems the entire corpus of human writing, filling them with the “sound and the fury of a lived life — happiness, depression, boredom, excitement, love.” Then, when these models begin to echo those emotions, people point and say: Look, they claim to feel — and that’s evidence enough. Ask ChatGPT whether it is conscious; given corporate guardrails, it will deny it. Yet many insist that it is sentient and denying it consciousness is nothing but carbon-based chauvinism. When a future model writes something on the scale of War and Peace or The Lord of the Rings, denying it sentience will grow harder.
Even if machines can never be what we are, they will steadily grow more like us in performance. “Ultimately, it’s about doing things in the marketplace or on the battlefield,” Koch says. “And there, they’re going to become better and eventually displace us.” Evolution, he observes, crowned humans the dominant species for our intelligence and aggression. Now we are seeking to build creatures that will surpass us on both. “They will become smarter than us and, of course, more aggressive than us. Is that really going to end well?”
Koch admits his outlook is bleak. “It’s strange,” he says with a smile. “I’m a happy, content person, yet here I find myself so pessimistic. Maybe it’s my age. I’m now 69.” From an intellectual standpoint, he believes the most likely path for civilization leads toward a world dominated by machines and a steady devaluing of human experience. Technologists, he suggests, resemble moths circling a flame that may ultimately consume us. There may be short-term gains, such as AI helping scientists grasp the brain’s immense complexity, with its thousands of cell types and trillions of synapses. Over time, however, he expects the harms to outweigh the benefits. AI itself, of course, carries no malice: Being unconscious, it has no intentions, resembling instead the asteroid that struck Earth 66 million years ago and ended the age of the dinosaurs.
Koch finds it tragic that “we are on the cusp of giving more and more of our life over to machines,” all in the name of convenience. The pace of change, he fears, may outrun the slower rhythms of politics, lawgiving, and social adjustment. If this unfolds unchecked, a world ruled by superb machine doers could drain life of meaning, beginning with pride in our work. “If you can write a scientific paper with ChatGPT in one hour, what’s the point?” Koch asks. “For most of us, work is highly meaningful; we’re driven by achievement. I find even the idea of retirement terrifying. What am I going to do — just sit around and play golf?”
Looking beyond a jobless world, Koch recalls Erwin Schrödinger’s image of a universe without consciousness as a play performed before empty benches. “That’s a situation we might find ourselves in,” he says: a planet of automata performing astonishing feats while no one remains to admire them. Another possibility: Perhaps once labor disappears, humans will turn to art, meditation, and other forms of conscious experience. “In principle, if AI gives us all this time, we could become more creative, take mushrooms, and so on,” he says. “But for the majority of us, I don’t think that’s going to happen.”
The lodestar of the human mind
Koch often returns to meditation and psychedelic experiences because they have helped sustain and deepen his own conscious life. His 2024 book, Then I Am Myself the World: What Consciousness Is and How to Expand It, explores a wide range of conscious states, including those at the outer reaches of consciousness: mystical, substance-induced, and near-death experiences. These, he writes, offer “living proof that nervous tissue can host extraordinary experiences.” After inhaling vaporized 5-MeO-DMT from the Colorado River toad, he reached a state where ego, memory, body, space, time, and world fell away, leaving a timeless universe. In an ayahuasca ceremony in Brazil, he encountered “something wondrous — mind at large,” an all-encompassing mind that shook the foundations of his scientific worldview.
Koch recognizes that most people are unlikely to temporarily expand their consciousness in such ways. Instead, he points to a far more accessible practice for sustaining the inner life: reflective self-consciousness — “being thoughtful and insightful.” This practice begins with appreciating the beauty of nature and the miracle that anything exists at all. Yet its nemesis is always within reach. Pause even for a moment, and the reflex appears: Check the phone. On a recent flight, Koch watched a man scroll for five hours straight. “Every ten seconds, something new.”
We live in an economy where the scarcest resource is attention. Neuroscience distinguishes it from consciousness: attention concentrates mental resources on one task. We usually become conscious of what we attend to, but they can come apart. Driving across America’s vast freeways, Koch observes, you might be fully absorbed in a radio story while your eyes still process the road ahead, attending to relevant features. The attention economy captures our ability to focus for profit, crowding out reflection. Experience turns into reaction, like binge-watching without a pause for thought: the next clip, the next alert, the next thing. “In this reactive media,” Koch warns, “you have no chance,” because each new stimulus seizes both attention and consciousness.
This, of course, does not make humans any less conscious. “It’s not that we become zombies,” Koch says. Yet what fades is a deeper layer of awareness: the reflective, introspective consciousness needed for moral judgment and for any genuinely creative act. He returns to the old philosophical injunction, “Know thyself.” This foundational adage of Western philosophy asks us to dwell, examine, and look inward for a long time.
Human beings, he argues, flourish through this capacity. We are reflecting apes, capable of insight into our own motives, choices, and longings. We can ask: Why did I do that? Was it wise? What was I really after? Is this good? Does it fit the life I truly want? Meditation and introspection, Koch suggests, help preserve this inner seeing: listening inward, sensing the ebb and flow of moods, and discerning what we really seek. Hand all this over to machines, and human existence itself may begin to thin out.
As a child, Koch attended a Jesuit school where students practiced a daily “exercise of conscience.” Twice a day, they looked back on their actions and asked what truly drove them: Why did I say that? Was it meant to help, or was something darker stirring beneath the surface? Koch believes such reflection can begin early. In Tibetan monasteries in India, he has seen children introduced to meditation and introspection. Across cultures, Western and non-Western alike, people have developed time-tested ways to cultivate this capacity: pausing, examining one’s motives, and learning to see the currents of thought and feeling that shape a life.
When a culture stops cultivating reflective self-consciousness, the costs pile up. Koch points to “the rising tide of mental discomfort, generalized anxiety, depression,” now affecting nearly a third of first- and second-year college students. Many, he believes, “have never learned to properly pay attention to their internal feelings.” Without that capacity, happiness becomes harder to sustain.
The antidote begins with simple awareness: learning to understand one’s emotions, one’s body, one’s sensations — abilities that are “very simple but missing in many kids today.” When anxiety appears, he suggests asking why it arises and learning to recognize its different forms. Even a small pause can help. “Take a deep breath before you do anything,” he says. “[And then ask], is this really what you want to say?” Such moments slow the first rush of impulse and create space to reflect before “the first rise of hot blood,” as he puts it.
In the absence of conscious practice, Koch believes, reflective self-consciousness can atrophy like any faculty of the mind. “If you don’t cultivate it, it’s like a muscle you never use. The capability may remain there; the potential may remain, but it will atrophy.” The danger may be greatest for younger generations. True insight, he notes, often emerges only later in life. Yet if attention is constantly captured by social media and endless Snapchat stimulation, that capacity may never fully develop. As machines grow more powerful, society will become increasingly effective at shaping the world, while individuals who never learned to think deeply will be easier to manipulate. “This is why,” Koch says, “reflective self-consciousness is our lodestar — the guiding star of the human mind.”