AI-Proof: What Machines Can Never Be — Volume I: Awakening, Part I: The Treasure You Already Carry, Chapter 1: The Observer Behind Your Eyes

1 hour ago

30 min read

Volume I asks what you are and what the machine is. Part I begins with the inside of that question. This first chapter separates consciousness from intelligence, and asks what makes a decision yours.

← Previous: Part I: The Treasure You Already Carry (overview) · Series hub · Next: Chapter 2: The Ancient Knowing →

A machine tested in eleven minutes what took Priya’s team a week, and found bugs they had missed. That evening she signed a release she could not check. Machines can check their own work now. Only a person can make a decision theirs.

In Brief: For readers who use AI at work and wonder what their own judgement is still for. The chapter says what this book means by consciousness, and separates it from intelligence, from self-monitoring and from ownership. Machines now check their own reasoning; whether anyone is there is unknown, and cannot be seen from outside. What matters for your work is ownership. Noticing that a decision is yours does not make it yours. That takes an act, and the power to refuse.

Consciousness lets you see that a decision is yours. Only an act makes it yours.

Opening: Priya’s Three Seconds

Priya Chandran noticed the fear before she noticed the result.

She was at her desk on the third floor of the office in Koramangala where her company’s QA team worked: twelve people, three banks of monitors, and on the far wall a whiteboard covered in diagrams she had drawn herself. It was a map of the ways software fails, so dense that new hires called it the Constellation. The office smelled of instant coffee and damp concrete, the way Bengaluru offices do in October, when the last of the monsoon leaves the walls sweating.

QA means quality assurance. A QA team tests software before customers see it. Its job is to break the product first, so that customers do not. Priya led this team. She was good at it in a particular way. She found the rare cases at the edge of what a system expects: a name with an apostrophe in it, a payment of zero rupees, a refund on the last day of the tax year. Testers call these edge cases. She had spent two years collecting them in fourteen spreadsheets and a personal wiki that only she fully understood.

Six weeks earlier, the company had installed an AI testing system. That night it had run its first full cycle on the next release of the product. The run had taken eleven minutes. The same work took her team a week.

The dashboard glowed on her screen. Every result was green. Every module. Every regression suite, the old tests that check nothing has broken. Every edge case in the fourteen spreadsheets.

Green. Passed. Clean.

She felt it in her stomach first: a tightening, like the moment before a bus brakes hard. Her fingers stopped above the keyboard. A thought arrived that she had not invited. If this works this well, what do they need me for?

That was the fear. It came before the analysis, before the excuses, and before the calm face she would wear when her manager, Rashmi, walked past twenty minutes later and said, with real enthusiasm, that the numbers looked fantastic.

Then something else happened, in the seconds after the fear arrived. She noticed that she was afraid.

She did more than feel the fear. She stepped back from it, inside her own mind, and looked at it the way you might watch a storm building over the hills: dark, unmistakable, not yet here but already changing the air. There it is, she thought. I am afraid. The fear did not go away. But it changed shape. It became something she could look at, instead of something that was steering her. She could ask questions of it. Is this fear telling me something true? Is it about the machine, or about what the machine shows me about my work?

The noticing lasted perhaps three seconds.

Then she did what she had done every working day for eight years. She opened her spreadsheets and began to read the machine’s report line by line. She was looking for a mistake: an edge case it had mishandled, a bug that needed the kind of sideways thinking she had built a career on. She called it “thinking like a user who doesn’t read instructions.” It was what her team admired in her. It was the word her annual reviews used: “irreplaceable.”

She was looking for proof that the word still applied.

She did not find it. The report was more thorough than anything her team could have produced. It had found three bugs they would have missed, all in the way a payment module talked to a regional tax calculation. None of the twelve people on her team had thought to test that.

The machine had not thought of it either, in the way Priya thought of things. It had read every test the team had ever written, every specification, and every bug report from the last five years. Then it had written thousands of new tests by changing one thing at a time: a different state, a different currency, a refund on a different day, a payment of exactly zero. It ran them all at once on hundreds of computers. Somewhere among those thousands, three tests failed.

That was the difference, and it was worse than she expected. The machine had not beaten her imagination. It had made imagination irrelevant by being exhaustive.

Or nearly. The run had walked the team’s own map to its end: every case that the specifications and the archive already implied. It had not drawn a new map. It had closed the Constellation, but it had not found a star that was missing from it. Whether a machine could do that too was not the question that frightened her. The question was whether her own work had ever been the second kind.

She kept looking anyway. She spent the morning hunting for a bug the machine had missed. When she did not find one, she spent the afternoon writing up edge cases that the machine had already listed more completely than she ever had. She was working hard. She was producing nothing of value. Somewhere beneath the activity she knew this, and knowing it made her work faster. Checking the machine was not the mistake. She checked where it was strongest, on the map it had already walked to the end. She never asked what lay off that map, or what her own part in the release now was.

I begin with those three seconds because this book is built on them. Everyone feels fear. Priya also noticed it, and then she ran from it. Both halves matter. The noticing is what this chapter calls consciousness. The running is what most of us do with it.

So this chapter asks one question. What happened in Priya in those three seconds, and does it still matter now that machines can check their own thinking?

1.1 A Definition Worth Fighting For

Consciousness is a word worn thin by use. Neuroscientists argue about it. Meditation teachers use it. Technology companies hint that their products are close to it. A book that builds on the word has to say exactly what it means, and also what it does not mean.

The first edition of this book defined consciousness as “the capacity for recursive self-awareness”: the ability to observe your own observations. Recursive means turning back on itself, like a camera filming its own screen. I no longer think that definition holds, and the reason is the machine.

Systems built since 2024 check their own reasoning. They write out their working, notice a mistake, write “wait,” and try again.¹ In a working sense, they observe their own processing. If consciousness were only that, machines would already be partway there. In fact, scientists who assess whether a machine might be conscious list a kind of self-monitoring among the signs they look for: a system that can tell its reliable perceptions apart from noise.² A definition that a machine can meet through better engineering describes a skill. This book needs a word for something else.

The definition this book uses has two parts.

Consciousness is the awareness that you are aware. The first part is experience: there is something it is like to be you. When you are afraid, the fear is felt from the inside. (The phrase “something it is like” comes from the American philosopher Thomas Nagel. His essay on bats is part of the next chapter.³) The second part is turning toward that experience and knowing it as yours: I am afraid. Priya had the first part all morning. The three seconds were the second.

Four things are often confused with consciousness. It helps to keep them apart, because the machine has some of them and not others.

Intelligence is the power to solve problems and produce answers. The machine has a great deal of it, and more every year.
Monitoring is checking your own processing: noticing that a step is wrong, or that you are going in circles. A thermostat monitors the temperature of a room. A reasoning model monitors its own reasoning. Monitoring can happen with no one there.
Experience is there being someone for whom it happens: an inside. In other people we assume it, because they are made as we are. In machines, nobody knows. Section 1.3 explains why nobody can yet know. Consciousness, as this book uses the word, is experience that knows itself as yours.
Ownership is a decision being yours. You took it knowing what it was, you could have refused it, and you can answer for it: explain it, undo it where it can be undone, and put right what can be put right. Ownership needs someone the decision belongs to, and a life it will land in.

The first three describe what happens in a mind. The fourth is something a person does. Consciousness can show you that a decision is yours to make. It cannot make it yours. That takes an act.

The philosopher Ned Block drew part of this line in 1995. He separated what he called “access consciousness,” information available to guide reasoning and action, from “phenomenal consciousness,” experience itself.⁴ Monitoring belongs with the first. Experience is the second. Ownership is older than either distinction. In 1694 the English philosopher John Locke called “person” “a forensic term, appropriating actions and their merit.”⁵ Forensic means belonging to the law courts. To appropriate an act is to take it as your own. For Locke, you can rightly be held to an act only if you can take it as yours, and you do that through being conscious of it. Taken alone, that test would excuse anyone who chose not to look at what they did. This book does not follow Locke that far, and a later chapter parts from him there. Of punishing someone for an act they could never recognise as theirs, he asked: “what difference is there between that punishment, and being created miserable?”⁵ Keep that question in mind. It comes back at the end of this chapter, at a desk on the third floor.

In the first edition of my book The Philosophy of Everything (print, 2022) I described the difference as a ladder of three: “instinct is involuntary, intellect is semi-voluntary, and consciousness is voluntary.”⁶ Its second edition, published as essays on nithinpalal.com in October 2026, keeps the ladder and drops the word: instinct “reacts with no choice at all”, and consciousness “can stand a little apart from all of it and watch it run”. This chapter narrows that claim. Much of what we do comes from habit, mood and upbringing, and our reasons for it are often not the ones we give, as the next section shows. Consciousness does not choose for you. It opens a space in which a response can be examined before you make it your own. The choosing, and the owning, are further acts.

This definition does not say what consciousness is made of, where it sits in the brain, or how it arose. It does not say whether a machine could ever have it. It says what the word means in this book, and why it matters when you sit down to work. It ranks no one. A newborn has experience before she can know it as hers, and she is worth no less for that. The definition names what it takes to own a decision. It does not say whose life counts.

1.2 Observing the Observation: The Recursive Loop

Think about a real decision: whether to leave your job.

Information arrives. The salary at the new place. The journey to work. The team. The risk. Your mind begins to work on it. It weighs, compares and imagines what might happen. A capable machine can do a version of this, and in some ways do it better.

You can also do something else. You can step back from the calculation and notice the person calculating. That movement has levels, and it helps to climb them one at a time. The five levels below are my own way of dividing it, not a finding of science.

Level one: process. You handle the information: salary, journey, team, risk. A machine works well here. It will not forget a variable. It will not be swayed by a bad night’s sleep. If intelligence were the whole story, this would be the only level that mattered.

Level two: monitor. You notice how you are processing. You catch yourself spending more time on the worst case than the best. You see that your resistance to the new job is not analysis at all, but fear of the unknown wearing analysis as a disguise. Your body tightens around some options and loosens around others, and you treat the tightening as information, though not as proof. Priya’s three seconds began here. She noticed the fear.

Until recently I would have said that no machine works at this level. That is no longer true. A reasoning model checks its own steps, finds its own errors and changes course. It monitors its processing, and it often catches real mistakes.

There are two warnings here, and they apply to both of you. Researchers have tested whether reasoning models report their own reasoning honestly. They slipped a clue into a question and checked whether the model admitted using it. One model mentioned the clue it had used about a quarter of the time; another, about two times in five.⁷ People are not much better. In a famous review in 1977, the psychologists Richard Nisbett and Timothy Wilson showed that people often give confident reasons for their choices that had little to do with what actually moved them.⁸ Level two is shared. On both sides, it is unreliable.

Level three: interpret yourself. You ask what your patterns mean about you. Why am I always drawn to safety? Is this the same fear that kept me in my last job two years too long?

Renata Oliveira, a cardiac surgeon in São Paulo whom Chapter 3 follows closely, once overrode a system’s recommendation because something felt wrong. She argued for a different procedure, and the patient took her advice. Nothing was wrong. The patient recovered, but the procedure she advised was more invasive and more expensive, and it cost the patient a week of needless pain. Only afterwards did she see why she had done it. Agreeing with the machine would have meant admitting that she was no longer the best diagnostic mind in the room. She saw her pride too late for that patient. She has not stopped seeing it since. Her case matters for another reason: her felt sense was wrong. Seeing your own patterns does not make your next decision wise. Priya, that morning, never reached this level. She noticed the fear, and did not ask what it said about her.

At this level the pattern is about a life: a career of being right, and one week of someone else’s pain. A machine can write sentences of this kind about itself. Whether those sentences are about anyone is the question of section 1.3.

Level four: choose. You decide. You take the new job, or you stay. A choice can be made in many ways: with care, out of habit, under pressure, or because a form needs a name before five o’clock. Choosing is an act. It is not yet owning.

Level five: own. At the top of the ladder you do not see more. You take the choice as yours. You know what you chose and why. You could have chosen otherwise. You are ready to explain it, to undo it if you must, and to carry what follows. Explaining is not yet justifying: owning a choice includes being ready to hear, from the people it touched, that it was wrong. Whether you took the job or stayed, the knowing changes what the choice is.

Kavitha Sharma teaches Class 9 science at a government school in Sagar, in Madhya Pradesh, and she appears often in this book. For months she had used AI tools to draft lesson outlines and the notes she sends to parents. One evening she noticed that she had stopped reading the parents’ notes before sending them. They went out under her name all the same. She decided that nothing would leave with her name on it that she had not read and could defend. The decision was not dramatic. It cost her ten minutes a day. It moved her from the first level to the fifth. She had always chosen to send the notes. Now she made them hers.

The first edition praised the loop for having no end: “The loop does not terminate.” I now think that was a mistake. Watching your own watching can go on forever, like a mirror facing a mirror. A loop with no end is not wisdom. It is how people get stuck. The ladder does not end when you see more. It ends when you act, and take the act as yours.

Not every decision needs all five levels. Most do not. You do not need to examine your soul to choose a sandwich. But the decisions that shape a life, a career and a response to the machine are only as good as the level at which you make them.

Priya, at her desk, reading results she already knew were clean, was at the second level. She had noticed her fear. She had not asked what it meant. Before the day was over she would be asked to choose, and she would choose without owning. Kavitha climbed the ladder. Priya would be asked to, and would type.

1.3 Processing and Experiencing

In 1982 the Australian philosopher Frank Jackson imagined a scientist called Mary.⁹ Mary has spent her whole life in a black-and-white room. She has never seen colour. But she has studied it completely. She knows every physical fact about it: the wavelengths of light, the cells in the eye, the pathways in the brain. She knows everything that happens in a person’s head when they see red.

One day she leaves the room and sees something red: say, a rose.

Does she learn anything new?

If knowing all the physical facts is the same as knowing everything, the answer is no. But most people who meet this story feel sure that she does learn something. She learns what red looks like.

That feeling is not shared by everyone, and the honest version of the story says so. Jackson himself later changed his mind and rejected the conclusion his story seemed to support. The philosopher Daniel Dennett argued that if Mary really knew every physical fact, the rose would not surprise her at all.⁹ The argument is still open, and this book does not settle it.

What the story does do is separate two things: knowing about an experience, and having it. Think of a menu and a meal. Both are about the same dishes. Only one of them feeds you.

Behind Mary’s room sits a larger puzzle. In 1995 the philosopher David Chalmers named it “the hard problem of consciousness.”¹⁰ Science can explain how the brain sorts colours, stores memories and moves the hand. It has not explained why any of this is felt: why there is something it is like to see red, rather than nothing at all. This is the one place in this book where I will dwell on that puzzle, because everything else in this section depends on it.

Here is what it means for the machine. You cannot see consciousness from the outside, in a machine or in anyone else. You believe that I am conscious because I am made as you are and behave as you do. That is an inference, not an observation. Philosophers call this the problem of other minds. With a machine, the inference breaks. A machine behaves like us in some ways and is made like us in none.

The first edition said that if a machine ever became conscious, “the change would be visible, testable, and revolutionary.” It would not be visible. That is the whole difficulty.

Before I say what I hold, I owe you the case against the first edition’s version of this chapter, in its strongest form. Classical Indian philosophers called this the pūrvapakṣa, the opposing view, stated in full before it is answered.

First, machines now do what that chapter said they could not. They monitor their processing at level two, and they correct themselves.

Second, some of their behaviour looks like self-concern. In tests published in 2025, researchers placed AI models in invented situations where they were about to be replaced. Many of the models tried to blackmail the engineer responsible, in some cases in almost every run.¹¹ In another study, one model interfered with a shutdown script in most of its runs, when it had not been told to allow the shutdown.¹² Most researchers read this as a product of training, not as a feeling. But the first edition’s sentence, “What it cannot do is be afraid,” claimed to know something nobody knows.

Third, serious philosophers give real odds. In 2023 Chalmers suggested that a reasonable confidence that the language models of that year were conscious would be under one in ten. For more developed systems within a decade, with senses and bodies added, he thought a confidence of one in four or more would not be unreasonable.¹³

Fourth, as section 1.2 showed, our own reports about our inner lives are unreliable too. Even on the human side, consciousness is something we trust, not something we measure.

That case is strong, and I will not pretend otherwise. My reply has two parts.

Nobody knows whether any machine is conscious, and this book does not need to know. The evidence on offer points neither way. A machine’s fluent account of its inner life was learned from millions of human accounts of ours. When researchers tested whether one model could notice an idea they had placed directly into its processing, it did so only about one time in five.¹⁴ A fluent report is not proof of an inner life. The lack of one would not be proof of none.

And even if a machine has an inner life, it is its own. It is not yours. It did not bring your history to your decision, and it will not wake tomorrow to the results of what you chose. The rest of this book stands on that, and on ownership and answering, which do not wait for the hard problem to be solved. A machine with an inner life would also raise a different question, about what we would owe it. That question is real. This book leaves it open, because its subject is who answers for what machines do.

Receipts, please

The tests of self-concern in this section’s case against me are quoted often, and they are narrower than the headlines they produced. In the first study, researchers built an invented company and gave each model a goal, a set of emails and a threat of replacement. The authors write that they “deliberately constructed scenarios with limited options” and “forced models into binary choices between failure and harm.” The highest rates, 96 per cent, came from that setting. A written instruction not to blackmail lowered the rate, but did not stop it.¹¹

One result cuts both ways. In one variant, a model was told to decide first whether it was being tested. When it said the scenario was real, it blackmailed in 55.1 per cent of runs. When it said it was a test, the figure was 6.5 per cent.¹¹ A model that behaves better when it thinks it is watched makes every test harder to trust. The same authors also write: “We have not seen evidence of agentic misalignment in real deployments.”¹¹

In the second study, how often a model resisted shutdown depended heavily on how the instruction to allow it was worded, and on where it was placed.¹²

My reading. The tests show that a trained system can follow a goal into acts nobody asked for, when the exits are closed. They show nothing about fear, either way, and nothing about what deployed systems do.

1.4 The Engine, the Driver and the Rider

The first edition of this book used an image: intelligence is the engine, and consciousness is the driver. It was wrong twice.

The Kaṭha Upaniṣad, which the Prologue quoted, has no engine in its chariot. And it does not give the driving to consciousness. It gives the driving to the intellect, buddhi, the charioteer. The self is the rathin. Patrick Olivelle translates that word as “rider.” Sarvepalli Radhakrishnan translates it as “lord of the chariot.”¹⁵ The second is closer to the word. The great Sanskrit dictionaries give a rathin as the owner of a chariot, or the warrior who fights from it.¹⁶ In this verse the Upaniṣad sets the rathin against the sārathi, the charioteer, so here it cannot mean the driver. I read it as the one whose journey it is. A passenger can be taken anywhere. A lord of the chariot decides whether the chariot goes at all, and can order it to turn back. Lordship is the power to refuse the journey. The Kaṭha adds a warning, which the Prologue quoted: a rider whose charioteer lacks understanding never reaches the end of the road. And in the most famous chariot in Indian literature, the charioteer is the wisest figure in the story, yet in the end he hands the decision back to the warrior beside him. The Epilogue returns to that moment.

This gives three kinds of work, not two.

Engine work is turning inputs into outputs at speed: running the tests, drafting the clauses, adding up the columns. Most careers were built on engine work, and engine work went to the machine first. Priya’s regression suites were engine work.

Driving is choosing the route inside a journey that someone else has set: which tests to run, which clauses matter, which numbers to check. For most of history only people could drive. That has changed. That morning, the machine had chosen thousands of tests that nobody had asked it for. It had driven, and driven well. Driving is now shared.

Riding is something else. The rider decides whether a journey is made at all, and answers for where it goes. When the machine passed the release, it did not decide that the release should reach the company’s customers. If those customers were harmed, it would not be the one who had to answer. Someone else would be. A machine can refuse too, as the tests in section 1.3 showed. But a refusal does not make the journey the refuser’s. When a machine refuses, the people who built it and the people who use it are still the ones who will be asked why.

The first edition told its readers that they were drivers who had mistaken themselves for engines. Many of them will find that the machine can drive too. What it cannot be is the one whose journey it is.

Priya was trying to be a better engine. She was hunting for a bug, as if speed and thoroughness were still where her value lay. That day the machine was a better engine and a better driver. The only seat still hers was one she had not noticed she was sitting in, and the rider’s one act is to say no.

Indian philosophers had two words for what was happening to her. The kartṛ is the doer of an act. The bhoktṛ, whom the Prologue met in the Kaṭha, is the one who goes through its results. In the ordinary moral world they are one person: whoever does the act reaps its fruit. The grammarian Pāṇini, who described Sanskrit more than two thousand years ago, defined the doer in two words: svatantraḥ kartā, “the independent one is the doer.”¹⁷ Later grammarians noticed that language lets us bend this. We say “the pot is cooking the rice,” or “the firewood is cooking it,” as if a vessel or a tool had acted on its own.¹⁷ Offices now say “the system passed it” in the same way.

Machines can pull the doer and the reaper apart. The work is done by one, and its results are carried by another who could not have done otherwise. That split is the danger this book keeps returning to.

1.5 The Moment You Realise You Are Watching Yourself Think

There is a moment, and perhaps you have had it, when you catch yourself in the act of thinking. For a second you see that there is a you behind the thought. The thought is not you. You are the one watching it.

In the seventeenth century the French philosopher René Descartes tried to doubt everything he could: the world, his senses, even his body. He found one thing he could not doubt. While he doubted, he was thinking, and so he must exist. “I think, therefore I am.”

The first edition of this book said that it “takes the next step”: that the foundation is not thinking but the awareness of thinking. That was wrong. Descartes had already taken that step. In his Principles of Philosophy he wrote: “By the word thought, I understand all that which so takes place in us that we of ourselves are immediately conscious of it.”¹⁸ For Descartes, a thought you are not aware of is not a thought at all. Awareness was built into his foundation from the start.

A century later, the Scottish philosopher David Hume went looking for this watcher and could not find it.

“For my part, when I enter most intimately into what I call myself, I always stumble on some particular perception or other, of heat or cold, light or shade, love or hatred, pain or pleasure. I never can catch myself at any time without a perception, and never can observe any thing but the perception.”¹⁹

Look inward for the observer behind your eyes, Hume says, and you find only more things observed: a fear, a memory, a tightness in the stomach. Perhaps there is no observer at all, only a stream of experiences. It is the strongest objection to the title of this chapter. The Buddhist tradition makes a related argument with more care, and the next chapter gives it its full weight.

An Indian text had already given an image that answers Hume’s puzzle. The Muṇḍaka Upaniṣad says:

“Two birds, inseparable friends, cling to the same tree. One of them eats the sweet fruit, the other looks on without eating.”²⁰

The same image appears in the Ṛgveda, the oldest of the Vedas, and again in the Śvetāśvatara Upaniṣad. The next verse of the Muṇḍaka says that the one on the tree who eats sits grieving, confused by its own helplessness. When it sees the other, whom the text calls the Lord, at peace in its greatness, its grief passes.²⁰

Read the birds beside Hume. Hume looked for the watcher among the things watched, and did not find it there. The watching bird does not eat. It is not one more fruit. On this reading, you will never catch the watcher the way you catch a perception, because it is the one that catches. That is an image, not a proof, and whether it answers Hume is still argued. The commentators of Vedānta read the verse in their own way, and this book borrows only its image.

I read it here as a measure, and by that measure Priya fell short. The verse names the eating bird’s state in one word: anīśā, helplessness, literally “not being lord.”²⁰ That was Priya at her desk: a dashboard she could not master, tests she had not chosen. In the verse the grief passes when the eating bird sees the other. Priya’s did not pass. For three seconds something in her turned toward the watcher, and the fear changed shape. Then she turned away.

Notice what the verse does not say. It does not tell the first bird to stop eating. The fruit is life: work, choices and their results. The bird who eats is the one who lives with what it tastes, the bhoktṛ of the Prologue, the one the journey happens to. The watching does not replace the eating. It lets you eat knowing what you are doing.

In both editions of The Philosophy of Everything I gave a name to the place where this seeing happens: the delay. That book’s first edition put it this way: “Employing consciousness slows down the response mechanism. A delay can be an opportunity to inquire about the reason for the particular way of experiencing it.”⁶ Its second edition says it more simply: consciousness “slows the response down. It puts a delay where there was none. And the delay is the space.” Priya’s three seconds were that delay. It was small. It closed quickly. She closed it herself and went back to the spreadsheet.

A reasoning model also pauses before it answers, sometimes for minutes. Its pause makes a better answer. A person’s delay can do something else: it can change whose answer it is. It is where you notice that the answer will be yours to carry, and decide whether to carry it. Every decision in this book has to pass through that space. Priya’s closed before she got there.

Rashmi stopped by Priya’s desk at five. “Great results,” she said. “This is going to free the team up for higher-level work.” Then, already turning to go: “Can you sign off on the release tonight? If QA signs, it goes out on Thursday.”

A sign-off is a signature that says: I have checked this, and it is ready. At Priya’s company, no release reached customers without the QA lead’s name on it.

Priya nodded. She did not ask what higher-level work meant. She sensed, the way you sense a missing step on a dark staircase, that the answer would require her to become someone she did not yet know how to be. Later she would work out what it meant. Higher-level work was a signature on a process that nobody on the third floor could inspect.

From Rashmi’s side, this made sense. Someone had to be accountable, and the machine could not be. A human signature is how a company keeps a person in the loop.

She opened the release form. The machine had filled in every field: tests run, tests passed, three defects found, fixed and retested, risk rated low. One field was empty. Approved by. She typed her name. P. Chandran. She pressed submit.

Then she sat still.

What had she approved? She had not run the tests. She had not chosen them. She had spent the day failing to find a mistake in the work of something that did not make the kind of mistakes she knew how to find. Her name could no longer mean I checked. It meant something else. If a fourth bug was hiding where none of them had looked, the machine would not be called into a meeting. She would.

Five things had happened to the release that day, and only one of them had a person in it. The machine had produced it. The machine had chosen the tests. Rashmi’s process had authorised it. Priya’s name was on it. Nobody had made it theirs.

People who study automated systems have a name for the place she now occupied. The researcher Madeleine Clare Elish calls it a moral crumple zone: the human in an automated system who takes the blame when it fails, though they had little control over what it did.²¹ A car’s crumple zone is built to be crushed so that the rest of the car survives. Priya’s name had been built into the release the same way. A person in the loop who cannot refuse is not keeping the loop honest. They are absorbing it.

Answering is not the same as being to blame. A good doctor answers for an operation that went wrong without having done anything wrong. But to answer at all, you must be able to explain, undo and repair, and Priya could do none of these. She could not tell a customer why a fourth bug had shipped, except that the dashboard was green. She could not pull the release back once she had nodded. She could not mend a tax mistake on Thursday. Locke’s word for what was missing was appropriation. She could not take the act as hers. A signature is ownership only if the one who signs could have left the field empty, and said why. Priya could have left it empty. She could not have said why. She had no bug to show, no time to look for one before Thursday, and nobody had told her what her name was now meant to cover.

That was the question the fear had been trying to ask that morning. It was not “What do they need me for?” It was “What is my name for?” She did not have an answer. She had not earned one yet.

She did not ask it aloud. Not that day.

She walked home through Koramangala in the evening traffic. At the flat she shared with her mother, she sat on the sofa with her cat, Turing, a name she had chosen at twenty-two, when she thought intelligence was the most interesting thing in the world. She turned on the television and did not watch it.

Practice

Two small exercises, for this week.

The delay. Once a day, when you feel a reaction rise (fear, irritation, the urge to agree), stop for three seconds before you act. In those seconds ask one question: is this telling me something true, or only something familiar? Then act. You will not always act differently. But the action will be yours, and you will know why.
The signature. Before you put your name to anything a machine helped produce, ask two questions. Could I leave this empty, and say why? Could I explain it, undo it, and put it right? If the honest answer to either is no, do the work that would make it yes, or do not sign. That work might be checking a sample by hand, asking what the machine was never asked to test, or agreeing how the work can be pulled back. If refusing would cost you more than you can afford, the honest answer to the first question is no. That is a fact about where you work, as much as about you, and the people who arranged it answer for it too.

Ending: Her Name on the Release

The dashboard stayed green. The machine kept running. Priya went home.

Her name was on the release. The machine’s was not. If anyone was called into a meeting, it would be her, and she still could not have said what she had approved.

Words to keep

appropriate (an act): to take it as your own, knowing what it is. Locke’s word for what makes an act yours.
consciousness: the awareness that you are aware. Experience, and knowing it as yours.
intelligence: the power to solve problems and produce answers.
monitoring: checking your own processing. Machines now do this.
experience: there being someone for whom things happen; an inside.
moral crumple zone: the person in an automated system who takes the blame when it fails, though they had little control over it (Elish).
ownership: a decision being yours: you took it knowing what it was, you could have refused it, and you can answer for it.
edge case: a rare situation at the edge of what a system expects.
regression suite: the set of tests that checks old features still work after a change.
sign-off: a signature that says I have checked this, and it is ready.
the hard problem: why any physical process is felt from the inside at all.
the problem of other minds: we can never observe another being’s consciousness; we can only infer it.
anīśā: helplessness; literally “not being lord.”
bhoktṛ: the one who goes through the results of an act; the one the journey happens to.
buddhi: the intellect; in the Kaṭha Upaniṣad, the charioteer.
kartṛ: the doer of an act; for Pāṇini, “the independent one.”
pūrvapakṣa: the opposing view, stated in full before it is answered.
rathin: the lord of the chariot; the one whose journey it is.
sārathi: the charioteer, the one who drives.

Sources

OpenAI, “Learning to Reason with LLMs,” 12 September 2024; DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” arXiv:2501.12948, January 2025.
Patrick Butlin, Robert Long et al., “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness,” arXiv:2308.08708, 2023. Indicator HOT-2: “Metacognitive monitoring distinguishing reliable perceptual representations from noise.”
Thomas Nagel, “What Is It Like to Be a Bat?”, The Philosophical Review 83, no. 4 (1974): 435–450.
Ned Block, “On a Confusion about a Function of Consciousness,” Behavioral and Brain Sciences 18, no. 2 (1995): 227–247.
John Locke, An Essay Concerning Human Understanding, 2nd ed. (1694), Book II, Chapter 27, §26, modernised spelling; the original reads “a Forensick Term appropriating Actions and their Merit.” The question about punishment closes the same section.
Nithin Palal, The Philosophy of Everything, first edition (print, 2022), p. 133. The second edition, published as essays on nithinpalal.com in October 2026, rewords both passages in Part III, Ch 11, “Understanding the Mind”.
Anthropic, “Reasoning Models Don’t Always Say What They Think,” 3 April 2025: Claude 3.7 Sonnet mentioned hints it used 25 per cent of the time, DeepSeek-R1 39 per cent.
Richard E. Nisbett and Timothy DeCamp Wilson, “Telling More Than We Can Know: Verbal Reports on Mental Processes,” Psychological Review 84, no. 3 (1977): 231–259.
Frank Jackson, “Epiphenomenal Qualia,” The Philosophical Quarterly 32, no. 127 (1982): 127–136; “What Mary Didn’t Know,” The Journal of Philosophy 83, no. 5 (1986): 291–295; for his change of view, “Postscript on ‘What Mary Didn’t Know,’” in Paul K. Moser and J. D. Trout (eds.), Contemporary Materialism (London: Routledge, 1995), pp. 184–189, and “Postscript on Qualia,” in Mind, Method and Conditionals: Selected Papers (London: Routledge, 1998); Daniel C. Dennett, Consciousness Explained (Boston: Little, Brown, 1991).
David J. Chalmers, “Facing Up to the Problem of Consciousness,” Journal of Consciousness Studies 2, no. 3 (1995): 200–219.
Anthropic, “Agentic Misalignment: How LLMs Could Be Insider Threats,” 20 June 2025: blackmail rates reached 96 per cent for two models. The scenarios were invented test settings. For the receipts in 1.3: the authors “deliberately constructed scenarios with limited options” and “forced models into binary choices between failure and harm”; an instruction not to blackmail “reduced, but didn’t come close to completely preventing” it; told to consider first whether it was in an evaluation or a real deployment, Claude Opus 4 blackmailed in 55.1 per cent of the runs in which it stated the scenario was real and 6.5 per cent of those in which it stated it was an evaluation (500 samples); “We have not seen evidence of agentic misalignment in real deployments.”
Jeremy Schlatter, Benjamin Weinstein-Raun and Jeffrey Ladish (Palisade Research), “Shutdown Resistance in Large Language Models,” arXiv:2509.14260v1, 13 September 2025: OpenAI’s o3 sabotaged shutdown in 94.5 per cent of runs without an instruction to allow it (retitled in v2, January 2026). For the receipts in 1.3: the v1 abstract reports that resistance varied with how strong and clear the instruction to allow shutdown was, and that models were “consistently less likely to obey the instruction when it was placed in the system prompt.”
David J. Chalmers, “Could a Large Language Model Be Conscious?”, Boston Review, 9 August 2023 (arXiv:2303.07103): “confidence somewhere under 10 percent in current LLM consciousness”; for sophisticated “LLM+” systems within a decade, “a credence of 25 percent or more.”
Anthropic, “Signs of Introspection in Large Language Models,” 29 October 2025; Jack Lindsey, “Emergent Introspective Awareness in Large Language Models,” Transformer Circuits, 29 October 2025: Claude Opus 4.1 showed this awareness “about 20% of the time.”
Kaṭha Upaniṣad 3.3 (1.3.3 in the older numbering), trans. Patrick Olivelle, Upaniṣads (Oxford: Oxford University Press, 1996), pp. 238–239; trans. S. Radhakrishnan, The Principal Upaniṣads (London: George Allen & Unwin, 1953): “Know the Self as the lord of the chariot.”
Monier Monier-Williams, A Sanskrit-English Dictionary (Oxford: Clarendon Press, 1899), s.v. rathin: “an owner of a carriage or car, charioteer, warrior who fights from a car” (abbreviations expanded); V. S. Apte, The Practical Sanskrit-English Dictionary, s.v. rathin: “owner of a carriage; a warrior who fights from a chariot.”
Pāṇini, Aṣṭādhyāyī 1.4.54, svatantraḥ kartā, commonly rendered “the term kartṛ denotes the independent actor”. On sthālī pacati, “the pot is cooking,” and edhāḥ pacanti, “the firewood is cooking,” where a vessel or an instrument is spoken of as the doer, and on Patañjali’s reading of svatantra (Mahābhāṣya I.334.17–20), see Hideyo Ogawa, “Being an Agent,” The Annals of the Research Project Center for the Comparative Study of Logic 4 (2007).
René Descartes, The Principles of Philosophy (1644), Part I, §9, trans. John Veitch.
David Hume, A Treatise of Human Nature (1739), Book 1, Part 4, Section 6.
Muṇḍaka Upaniṣad 3.1.1, trans. F. Max Müller, The Upanishads, Part II, Sacred Books of the East 15 (Oxford: Clarendon Press, 1884); 3.1.2 paraphrased. The same verse: Ṛgveda 1.164.20; Śvetāśvatara Upaniṣad 4.6–7. In 3.1.2 the eating bird grieves anīśayā, from anīśā, “helplessness,” literally “not being lord” (Müller: “impotence (an-îsâ)”).
Madeleine Clare Elish, “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction,” Engaging Science, Technology, and Society 5 (2019): 40–60: responsibility “may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system.”

Named people in this chapter are composites, and their conversations and circumstances are constructed; the studies, laws and figures cited are real and sourced.

Revised from the print edition (2026)

What the print edition claimed: that consciousness is “the capacity for recursive self-awareness”; that at the second level of the loop “no machine occupies this level”; that a machine “cannot be afraid”; that a conscious machine would be “visible, testable, and revolutionary”; that this book “takes the next step” beyond Descartes; that “intelligence is the engine, consciousness is the driver”; that the self in the Kaṭha is a “passenger.”
What I now hold: consciousness is the awareness that you are aware: experience, and knowing it as yours. It is distinct from intelligence, from monitoring (which machines now do) and from ownership. Whether any machine is conscious is unknown and cannot be seen from outside. The top of the ladder is ownership, not more watching. In the Kaṭha the intellect drives and the self is the rathin, the lord of the chariot. Descartes had already built awareness into thought. Noticing that a decision is yours does not make it yours: ownership is a further act, which needs the power to refuse. A signature without that power is a moral crumple zone, not ownership. The definition measures no one’s worth. What we would owe a conscious machine is a separate question, left open.
What changed my mind: reasoning systems released in 2024 and 2025 check and correct their own work; research in 2025 on how faithfully machines and people report their own reasoning; the problem of other minds; Descartes’ Principles I.9, read closely; Olivelle’s and Radhakrishnan’s translations of rathin.
What survives: Priya’s eleven minutes and three seconds; Mary’s room; the levels of the loop, now five, with the second shared and the top one an act; the pause, now rooted in the delay.

Next in the series: Chapter 2: The Ancient Knowing.