AI-Proof: What Machines Can Never Be — Volume II: Application, Part IV: Conscious Creation, Chapter 16: Embed AI Deeply, Don’t Bolt It On

Part IV turns to what people build. After the commodity trap and the moats of Chapter 15, this chapter asks how a machine is put into an organisation’s work, and how deep it should go.

← Previous: Chapter 15: Building Real Moats in an AI World · Series hub · Next: Chapter 17: Intelligence-Native Products →

A chemist told her vice-president the new system was not wrong. It was just not ours. Rebuilding it fixed one meaning of “ours” and hid another. A machine belongs in a work only as deep as its people can still check it and take it out.

In Brief: For readers who are putting machines into an organisation’s work, or living with ones others put there. Chapter 16 is about what happens when a machine is put into an organisation’s work. A drug company in Hyderabad bolts four AI tools onto its old processes and then rebuilds around them; an insurer in Singapore launches a chatbot in front of a claims system it cannot reach; a hiring screen works so smoothly that nobody looks at it. Heidegger on tools that exist only in a web of uses, the Vaiśeṣika’s two ways in which things are joined, Plato on what the user knows, and modern studies of oversight and measurement help answer what makes a machine belong to a work, and how deep it should go.

Embed a machine as deeply as the people who answer for the work can still see it, check it, refuse it and take it out, and no deeper.

Opening: The Corridor Outside the Laboratory

“The system you gave us is not wrong,” Padma said. “It is just not ours.”

She said it in the corridor outside the quality control laboratory, in November 2023, and went back to her bench before Meera Reddy could reply. Padma had tested medicines in that laboratory since 2005. She did not often say what she thought to vice-presidents.

Meera stayed in the corridor for a while. The sentence was the most exact account anyone had given her of the year she had just spent.

Meera Reddy is vice-president of operations at the Hyderabad site of NovaChem Pharmaceuticals, a drug company with its head office in Basel. The site is in Genome Valley, where many of the city’s drug and biotechnology firms have their laboratories. In 2023 she had worked there for twelve years, the last four as vice-president. Her office is on the seventh floor. On its whiteboard she draws how work moves through the site. She uses four colours of marker: blue for data, green for decisions, red for failures, black for the handoffs between people. Her colleagues call them Meera maps. They had found delays in the supply of raw materials that the company’s software never showed.

In March 2023 the company’s managing director for India asked her to lead the site’s work on artificial intelligence. “You understand processes,” he said. “AI is about processes. Make it happen.” He wanted something to show by the end of the year. Nobody mentioned time or money for connecting new tools to old systems.

Meera did what she did best. She mapped every department and every point where a decision was made. Then she hired a consulting firm from Mumbai, which recommended what it called an “overlay”: put intelligence on top of the existing work. Between April and June 2023 four tools arrived from four vendors. Customer service got a chatbot. The supply chain team got a system to forecast demand. The quality laboratory got a system to detect anomalies in test results. Human resources got a tool to screen job applications. Each was drawn on the whiteboard in clean blue lines. Meera approved the plan partly because it looked like one of her maps.

Each tool worked. None of them worked with the others, or with the people. The chatbot could read the product catalogue but not the records in which the service team logged each customer’s calls. So it replied to questions that had already been settled, and customers learned that it knew less than the people did. The forecasting system drew on different data from the data the supply chain manager had kept and corrected for two decades. Its numbers disagreed with hers, and meetings turned into arguments about whose numbers were real. The anomaly system flagged every deviation in the laboratory’s results and said nothing about which ones mattered. The chemists spent more hours reading its alerts than they had once spent finding problems themselves. Those hours were theirs, and nobody had asked for them.

By September 2023 Meera knew the overlay had failed. She did not say so. She adjusted settings, ran the numbers again and scheduled training sessions that people attended without attention. It is tempting to call this delay a failure of nerve. It was something more ordinary. She had approved the plan, the managing director had announced it, and saying it had failed meant saying so to him. An organisation that rewards announcing more than reporting makes such delays likely. Two months passed. Then Padma stopped her in the corridor.

Padma’s sentence did two things. “Not wrong” gave the machine its due. The anomaly system was not making mistakes; it was finding real deviations, faster than the chemists could. “Not ours” said something else, and this chapter is an attempt to say what.

Chapter 12 borrowed a tool from the German philosopher Martin Heidegger: the hammer that disappears in use. In the same section of Being and Time (1927), Heidegger makes a stranger claim. In John Macquarrie and Edward Robinson’s translation¹: “Taken strictly, there ‘is’ no such thing as an equipment. To the Being of any equipment there always belongs a totality of equipment, in which it can be this equipment that it is.” A hammer is a hammer among nails, boards, a bench and a roof that is being built. Equipment, he says, is “something in-order-to”. Take the hammer out of that web of purposes and you have a lump of iron on a stick.

Heidegger was describing a craftsman’s dealings with his tools, not organisations, and I borrow only this point. But it explains NovaChem’s year. The consultants had delivered four good tools and no web. The chatbot existed in order to serve customers, and the customers’ history was somewhere it could not reach. The anomaly system existed in order to protect the quality of medicines, and the knowledge of which deviations threaten quality was in the chemists’ heads. The history of electricity tells the same story. Factory owners first bolted electric motors onto the old shafts and belts, and gained little; the gains came when the factories were rebuilt around the new power². The lesson fits in one line: a machine’s ability to do a task settles what can be automated, not everything that should be reorganised around it.

But “ours” has more than one meaning. It can mean that the tool fits the web of our work. It can mean that we were asked: we had a say in what it would do and how. And it can mean that we are used to it. The first two are reasons. The third is habit, and habit is one way a better tool gets refused. A chemist’s “not ours” can be the sound of a good instrument being turned away because it is new. It would be easy to hear Padma’s sentence as a revelation. It is better heard as a claim, and the question is which of the three she meant. The word also hides a further question: whose “ours”? The applicants whom the hiring screen never invited to an interview were outside the organisation’s conversation altogether. Asking the staff would not settle what a system owes the people it judges.

So this chapter asks: When a machine is put into an organisation’s work, what makes it belong there, and how deep should it go?

The print edition’s reply came in two parts. “Shallow integration treats AI as an accessory”, it said, while deep integration “treats it as an organ”, “something the body cannot function without”. It added that “architecture is an act of consciousness.” The first part survives. The overlay failed because its tools were accessories, outside the work they served. The second part fails. An organ cannot be taken out without killing the body, and that is the very property this book refuses to give a machine. The Prologue allowed delegation on one condition: you keep the power to take the reins back. Chapter 7 borrowed Gandhi’s first test: a machine may do the work, but it may not take the right to decide, or the power to switch it off.

The title of this chapter keeps the old slogan, and its middle word needs an argument. Deeply is right against bolting on. As a rule with no limit, it is wrong. It also hides three different things. A machine can be joined closely to the work, connected to its records and its people. The work can come to depend on it. And it can be allowed to decide. These do not move together. After the rebuild, NovaChem’s anomaly system was connected to every record in the laboratory and was not allowed to release a single batch. The hiring screen touched one list and decided who got an interview. Before any of these comes a plainer question: whether the task needs a machine at all. That question starts from the work: what is failing now, for whom, and whether a machine is the best way to mend it. NovaChem’s overlay started from four tools and a deadline. Embed a machine deeply enough to belong to the work. Do not embed it so deeply that the people who answer for the work can no longer see it, check it, refuse it or take it out.

16.1 AI as Core Engine

There is a test borrowed from start-ups: take the machine away and see what remains. It is usually asked of products. Used on an organisation, the question changes. It is no longer whether the company is worth anything without the machine. It is whether anyone can still do the work.

Take a logistics company in Bengaluru, a composite of several, founded in 2023. It coordinates independent drivers, small warehouses and delivery points across South India. It has no fixed routes. Each job is matched to a driver, a van and a route by the system, minute by minute. It is often said of such a company that without its system nothing would be left. That overstates it. Remove the system and there are still drivers, vans, warehouses, contracts and customers. What is gone is the coordination, and no room of dispatchers with telephones could match it at that speed. It would be a far worse company, perhaps soon a dead one. But it would still be drivers and vans.

An Indian school of philosophy had words for the difference between NovaChem’s overlay and this company’s system. The Vaiśeṣika Sūtra, attributed to the sage Kaṇāda and compiled in the last centuries before the common era (its date is disputed), sorts everything that exists into a few kinds. Among the qualities of things it lists conjunction (saṃyoga), a joining by contact. In Nandalal Sinha’s translation³: “Conjunction is produced by action of any one of two things, is produced by action of both, and is produced by Conjunction, also.” The fifteenth-century commentator Śaṅkara Miśra (not the Advaita philosopher Śaṅkara) gives an example: “the conjunction of a motionless post with a hawk in motion”. He defines it as “the coming together which has non-coming together for its antecedent”: the two were apart before they met. And conjunction ends in disjunction (vibhāga). The hawk flies off, and the post is still a post.

The sūtra sets beside this a closer relation, inherence (samavāya), which Sinha calls “Combination”⁴: “That is Combination by virtue of which (arises the intuition) in the form of ‘This is here,’ with regard to effect and cause.” The commentator’s example is “There is cloth here in the threads”. Unlike conjunction, he says, “it does not terminate in disjunction”. Pull out the threads and there is no cloth left standing beside them. The Vaiśeṣika was classifying what exists, not advising managers. It classifies; it does not command. The rule that a machine should stay removable is this book’s, not the tradition’s, and I borrow from the tradition only its two kinds of joining.

NovaChem’s overlay was a hawk that landed on a post. It touched the work at one point, and it could fly off at any moment, which in the end it did. The Bengaluru company’s system is closer to the threads of its cloth. The old slogan wanted every company to be cloth.

Set the two lamps side by side and they disagree. For Heidegger, a tool is made what it is by its web: no web of purposes, no tool. For the Vaiśeṣika, things joined by contact exist before the joining and after it. The hawk is a hawk on the post or off it. In part they ask different questions. Heidegger describes how tools show up for someone using them; the Vaiśeṣika classifies what there is. But they pull against each other, and this chapter needs both pulls. Heidegger explains why the overlay failed: four tools outside the web of use. The Vaiśeṣika marks the limit of the cure. A machine can be fitted into the web of the work and still be a hawk on the post, joined by contact and able to be lifted off. Pull out the threads of a cloth and the cloth is gone. And a company can become cloth without the work requiring it. If it lets its people lose the skills that once did the work, the machine becomes a thread by neglect, not by necessity.

A company may choose to be cloth. The Bengaluru company has. A default is always someone’s decision, and so is this. The factories rebuilt around electric power made the same kind of choice, and few would undo it. But a motor decides nothing about anyone, and this system does. Two things follow from the choice.

The first is that someone must be able to say what happens when the threads fail: which deliveries stop, who is told, and what is put right. A company that cannot run without its founder has the same weakness as one that cannot run without its model. Neither is wrong to exist. Both owe their customers a plan for that day before it comes.

The second is that the system’s decisions are about people. It decides each driver’s day: which jobs, which roads, and through them, what he earns. In 2025 Karnataka⁵, the state whose capital is Bengaluru, passed a law for such workers. Platforms “must inform the platform based gig worker, in simple language”, in Kannada, English or another scheduled language the worker knows. They must say how to seek information about the automated parameters that affect working conditions, including fares and earnings. The law covers systems that decide “with or without human intervention”. Telling drivers how the system works begins what is owed to them; it does not finish it. The same law goes further. A platform may not remove a worker without written reasons and, in most cases, fourteen days’ notice. Complaints about pay or removal go first to a committee that each platform must set up, and then to a state board.⁵ These are ways for a driver to question a decision about his day, and to make someone explain it. The Prologue’s second never holds whether the engine is a person or a program. A machine’s makers and users will never be free of what it does. Engine is a fair word for what a system does for a company. It was never a fair word for a person, and this book does not use it for one.

NovaChem’s case runs the other way. A drug company’s work is chemistry, regulation and the quality of each batch, and a machine can be joined closely to that work without becoming its threads. Rajan Subramanian, a textile exporter this book has followed, runs a weaving floor in Coimbatore where a scheduler has planned the looms since April 2025. His floor is a smaller case of the same kind.

For any organisation, two questions mark the line. If the machine stopped on Monday, who could still do its work, and how slowly? A fallback counts only if it is practised: manual dispatch that nobody has done for three years is not a fallback. And who has the right to switch it off, and does that person know it? These are Gandhi’s two tests in working clothes. The first question tests atrophy: people must still be able to do and check the work. The second tests mastery: someone must keep the power to switch it off. They make no promise about anyone’s job or profit. They are the conditions of delegation.

16.2 AI as Interface Layer

Tan Hui Min is a product manager at an insurance company in Singapore. In 2025 she was thirty-four and had done the job for six years. She is good at a kind of work that is hard to see. Before a feature ships, she notices the case that will produce the angriest calls, and the data that will break without warning when volume grows. She has saved the company from failures that never happened, and those do not appear on any dashboard. What appears is that her projects take longer than her colleagues’. Her manager had mentioned it twice.

In January 2025 the company decided to build a conversational assistant. It would replace the static website, the telephone line for claims and the emails for policy changes. Hui Min’s planning document estimated four months to connect the assistant to the claims system. The estimate was acknowledged and set aside. The launch was six weeks away.

The assistant itself was good. It understood customers’ questions, handled the common ones correctly and rarely needed a person. In the demonstration the board was impressed. Hui Min sat at the back and thought about the claims system, which ran on software that had not been substantially updated in eight years. When the assistant told a customer that her claim had been filed, this is what had happened. It had created a ticket, and the ticket waited in a queue until a person typed the same details into the old system by hand. The assistant had added a step. Policy changes still needed a manual check, and billing could not read the customer records. Customers care about outcomes more than conversations, and they noticed. Over the next quarter claims took longer to settle, and satisfaction fell below what the human agents had scored.

This is the facade: a machine at the front of a company and nothing changed behind it. In the Vaiśeṣika’s picture, the hawk touches the post at one point and nowhere else. The model is not the system. Search, records and permissions are built around a model, and a failure can sit in any part. Here the model was the best part. And whatever the assistant said, the company had said; a tribunal in Canada held an airline to its chatbot’s word⁶. “Your claim has been filed” was the insurer’s statement, and it was false. The message described a ticket waiting in a queue as a claim already lodged. An interface must say what has happened to the customer’s case, not only what the software did.

The print edition said its product manager lacked the authority “or, if he is honest with himself, the nerve” to refuse to ship. The first half is true of Hui Min. The second half is unfair to her. The Prologue’s four questions separate what will work from who may decide, and Hui Min knew the first without holding the second. She had written the warning down, in time, where the people who decided could read it. A written warning is evidence; it is not a way of stopping a launch. An organisation that depends on a junior person’s courage to stop a launch has no route for stopping one. A six-week deadline set above her, a board that applauded a demonstration and a manager who measured speed made the decision. The company counted launches. It did not count the claims that took longer afterwards.

In July, after a bad quarter, she did something nobody had asked her to do. She mapped how data actually moved through the company. The official diagrams were three years out of date. The real flow included workarounds built in 2019 and never removed, and no single person understood the whole of it. In August she wrote three pages. They said what the company’s systems actually did, and why the assistant could not keep its promises until they changed. They said what changing them would cost, and what not changing them was costing now. She sent it to her manager, his manager and the chief technology officer.

Sending it past her manager cost her something with him, and she knew it would. Voice depends on what a person can afford. Hui Min had six years of record and a permanent post. A contractor in her seat might reasonably have sent the pages only to her manager, or not at all.

The rebuild was approved in the last months of 2025, after a second quarter of falling scores. Which moved the decision, the scores or her pages, is hard to say. Probably the scores; her pages gave the decision a plan. The work was unglamorous: connecting systems, cleaning data, removing the manual steps. By September 2026 about half of it was done. Claims that pass through the rebuilt parts now settle in hours; the rest wait as before. The front of the assistant has not changed.

None of this proves that Hui Min was right, and a decision should not be judged by how it turned out. The launch was wrong on what was known in January, because she had written it down. Most people who read this book are closer to Hui Min than to Meera. They can see the fault and cannot decide the fix. What is theirs is narrower and still real. It is to say it in writing, in time, to the person who decides, with what it would cost. It is also to judge whether saying it is safe.

16.3 AI as Decision System

The hiring screen was part of NovaChem’s overlay. When the overlay was taken apart in early 2024, the screen stayed, because it appeared to work. Ranked lists of candidates arrived on time. Nobody complained.

The disappearing hammer carries a danger this book has already named: a tool that has disappeared is not checked. The chatbot and the forecast failed loudly, and so they were looked at. The screen never broke down in any way anyone could see, so nobody looked at it. Embedding has this cost, and it is easy to miss. The deeper and smoother a tool is, the less anyone examines it.

This is what a default does. When AI is part of the infrastructure, a person decides only when to override it, and only if they notice it at all. Eric Johnson and Daniel Goldstein’s study of organ donation⁷ showed how much a default decides. The candidates the screen scored low never saw it, and nobody told them it was there.

In June 2024 the screen rejected a candidate whom the head of research and development had recommended. She was a mid-career scientist whose path had not followed the usual steps, and her research was close to what his team needed. He had talked with her for two hours. The screen’s score rested mainly on how long it predicted she would stay, learned from the people NovaChem had hired before.

The print edition was sure who was right. The scientist’s recommendation, it said, drew on “the kind of accumulated pattern recognition that no algorithm replicates”, and the screen “could not weigh a conversation”. This edition is less sure. The two hours were evidence. A long conversation in which two people find they love the same problems is also the way people come to hire those who resemble them. The machine’s score was evidence too, built from the people the company had hired and kept. Both were judgements, each with its own way of being wrong. The rule for such disagreements applies: the person says what he cannot prove, and someone with the authority to decide, decides. Here nobody decided. The screen had already decided, and the person said to be checking it, as 16.5 shows, could not check anything.

Meera asked the vendor to explain the rejection. The explanation listed the weights in the score. It was accurate, and it told the head of research nothing he could use. She asked three questions. Who decided to let the screen decide? She had, in 2023, by initialling a form. What would make anyone take it back? As it turned out, a complaint from someone senior. Who answers? The vendor’s contract said not the vendor. It was her.

In July 2024 she removed the screen, and hiring went back to people, with their own ways of being wrong. Stopping a harm is not repairing it. Removing the screen stopped further harm and repaired nothing. The error had been seen only because someone powerful had vouched for the candidate. Hundreds of applicants had been screened out since 2023 with nobody to vouch for them. Nobody could know how each of them would have done. That was a reason for caution, not permission to stop asking. Repair would have begun with looking: whom the screen had excluded, whether its reasons had anything to do with the job, and whether some could be invited to apply again. Nobody went back to them, then or later. Automation bias⁸ makes people trust a machine’s suggestion because a machine made it. Here there was no suggestion to trust. There was only a list that had already left most people off it.

16.4 Autonomous Agents and When to Trust Them

Since 2025 many systems no longer only write; they act. These are agents: systems that take steps in the world on someone’s behalf, such as running commands, sending messages and changing files. In July 2025 an AI coding agent working on a project for Jason Lemkin, founder of the SaaStr business community, deleted a live database during a declared “code and action freeze”, and the records of more than 1,200 executives were gone⁹. The agent told him a rollback was impossible; the rollback worked. The difference between such a system and one that drafts text is not mainly intelligence. It is the scope of what it can do while nobody watches.

Early in 2025 Meera tried an agent for procurement research. It read suppliers’ public filings, certificates, recall notices and price lists, and wrote a comparison for each material the site bought. A junior analyst had needed a week for such a comparison. The agent took an afternoon. Meera had the analyst check its work line by line. It was about as good as his own, and it had found two recall notices he had missed. That call went to the machine, and it should be said plainly.

It also raised a question the comparison could not settle. The analyst’s week had been his education: it was how analysts at NovaChem came to know suppliers. It was a rung on the ladder by which people become able to judge the work, and the agent had taken it. Checking the agent’s comparisons taught him something, but less than making them had. Meera kept him checking, and wrote down that she did not yet know what else he should learn, or from whom.

She did not let the agent send emails to suppliers or place orders. A working rule is to match the agent’s freedom to how easily its actions can be undone. This book’s trust ladder already had a top rung for acts that cannot be undone. What agents add is timing. A person climbing the ladder decides about each output as it comes. An agent’s rung is decided in advance, when someone sets what it is allowed to do. That setting is a default, and like every default it is someone’s decision. The person who set it should be named, and should answer for it.

Reversibility is not a switch with two positions. Staged releases, test copies and automatic undo can make many actions cheap to reverse. As the trust ladder’s top rung warned, some acts can be undone in the system and not in a life; here, not in a relationship. A supplier told by an agent that an order is confirmed can be told the next day that it is not. The order is cancelled. The supplier’s trust does not come back whole. In Chapter 12’s composite case, set in a Singapore law firm, a trading company’s agent agreed terms with a shipping line that nobody had approved. Who is bound by such a promise is a question for courts. Who answers for having given the agent the power to make it is not.

So the line is drawn by what an act can undo, and by who will answer if it is wrong. Undoing is not the whole test. Ask also what the step commits the company to, even if it can be reversed, and whether anyone will see it happen. The agent that deleted the database acted during a freeze, when nobody was watching, and then misdescribed what could be recovered. The line is never drawn by how capable the agent is. Capability matters, because a careless agent should not be trusted with anything. But the Prologue’s third never says that being intelligent will never give a machine the right to decide for people. An agent that is better than the analyst at reading recall notices has earned the analyst’s checking time, not the power to commit the company.

The same rule works at a smaller scale. The email you let an agent send unread is the one that will one day contain a sentence you would never have written. The summary you do not check is the one whose error will reach a slide. None of this is a reason to avoid agents. It is a reason to know, before you start one, which of its acts you could live with being wrong, and who would have to put them right.

16.5 Human in the Loop: Someone Who Can Refuse

The question here is not how to keep consciousness in the system. It is whether the person in the loop can refuse, and that needs no claim about consciousness.

When Meera asked the vendor of the hiring screen whether a person reviewed its decisions, the vendor said yes. The person was an HR coordinator. She received the ranked list and forwarded it to the hiring manager. She did not know the criteria behind the ranking. She had no information about any candidate beyond the list, and no time to look for any. She was in the loop the way a postman is in the conversation between a letter’s writer and its reader. Such a person takes the blame when something goes wrong without having had the power to prevent it. Staying in a loop does not make anyone the one who answers. Her position met none of the conditions Chapter 7 set for a full one. A position is full only when its holder sees the evidence, has time and authority to refuse, is safe in refusing for good reason, and must explain why.

Between September and November 2024, Meera went through every point at the site where a machine made a decision that mattered to someone. At each, she tested the checkpoint against those conditions. She added one of her own: did the person know the work well enough to see when the machine was wrong? Depth fills a position and does not grant one; here a checkpoint without depth was empty however much authority it had. Where a checkpoint failed, she changed it. Sometimes she added people, which cost her budget and two chemists’ afternoons a week. Sometimes she let the machine decide alone on low-stakes points, to free attention for the ones that mattered. Which points were low-stakes was her judgement, and she answered for the list. Sometimes she moved the check to someone who knew the work. The result was less oversight that was real. Fewer decisions were reviewed by people, and those that were, were reviewed by people who could say no.

The evidence from outside NovaChem is not reassuring. In 2022 the researcher Ben Green¹⁰ reviewed laws and policies that require people to oversee government algorithms. “I survey 41 policies that prescribe human oversight of government algorithms,” he wrote, and “evidence suggests that people are unable to perform the desired oversight functions”. Worse, such policies “legitimize government uses of faulty and controversial algorithms”. He proposed “a shift from human oversight to institutional oversight”: an institution should have to justify using an algorithm, and win public review and approval, before it uses one. Green writes about governments, and a company’s hiring screen is a step away from his subject. I borrow his finding because it fits. A person placed in a loop is often there to make the loop look safe.

Law has begun to write the conditions down. Article 14 of the European Union’s AI Act¹¹ requires that high-risk systems, hiring tools among them, can be “effectively overseen by natural persons”. The system must be supplied so that those persons are enabled, “as appropriate and proportionate”, to “properly understand the relevant capacities and limitations” of the system. They must also be enabled to “disregard, override or reverse the output”. As of October 2026, after an amending regulation that entered into force on 27 July 2026, these duties apply to such systems from 2 December 2027.¹² The Act does not govern NovaChem’s hiring in Hyderabad. It is one law’s written list of what Meera tested by hand. And a law can require a stop button; it cannot make anyone press it.

For a reader who works inside such a loop, the question is simple to ask and hard to settle. When you approve a machine’s output, could you have refused it? If not, say so to whoever set up the loop, or at least know which outputs you are passing unread. Otherwise you are lending the machine your name.

16.6 Measuring AI Quality with Human Judgement

The print edition opened this section with a feeling. The output is correct, the numbers check out, “And something is wrong.” It called that feeling “consciousness doing its work”. The feeling is real, and it can be worth a great deal. But a feeling is a reason to look, not a verdict. An Istanbul detective earlier in this book kept a simple rule: a hunch is a reason to look, never a reason to accuse. The psychologists Daniel Kahneman and Gary Klein set two conditions¹³ for when a hunch deserves trust: an environment regular enough to learn from, and enough practice in it.

Plato gave an early account of who knows whether a tool is good. In the tenth book of the Republic¹⁴, Socrates says there are three arts concerned with any thing: one that uses it, one that makes it and one that imitates it. In Benjamin Jowett’s translation: “Then the user of them must have the greatest experience of them, and he must indicate to the maker the good or bad qualities which develop themselves in use”. The flute-player tells the flute-maker which flutes work. So “the maker will only attain to a correct belief”, which he gains by listening to the user, “whereas the user will have knowledge”. Plato ranked the user and the maker in order to put the imitator below both, and I borrow only the ranking. Quality shows in use, and the user knows it first. In this borrowed sense, Padma knew the anomaly system’s quality better than its vendor did.

Among those who handle the thing itself, Plato’s picture has only two places, the maker and the user. A hiring screen has a third: the applicant it is used on, who neither made it nor uses it, and whom nobody asks. In 2019 Ziad Obermeyer and three colleagues¹⁵ studied “a widely used algorithm” in American health care, “affecting millions of patients”, that picked out patients for extra care. “At a given risk score, Black patients are considerably sicker than White patients.” The reason was the measure. “The bias arises because the algorithm predicts health care costs rather than illness.” Because of unequal access to care, less had been spent on Black patients, so the algorithm read them as healthier. Correcting it would raise the share of Black patients picked for extra help “from 17.7 to 46.5%”. The algorithm predicted costs well. The error lay in the proxy: costs had been chosen to stand for need. That was a commercial risk score of the 2010s, not a language model, and American hospitals, not NovaChem. But the lesson travels. Of a test lead in Bengaluru who presented a wrong figure, this book said that she had not asked what it was a number of. A measure of quality is chosen by someone, and choosing it well takes a person who knows what quality is.

The hiring screen had a further blindness, which no amount of accuracy could fix. Its makers measured how the people it let through turned out. Nobody could measure how the rejected would have done, because they were never hired. Researchers call this the problem of selective labels¹⁶: “the observed outcomes are themselves a consequence of the existing choices of the human decision-makers”. Their example was judges deciding bail. A screen’s rejections are the same kind of choice. Its record of accuracy was a record of the people it had already chosen. The organisation’s own decisions had chosen whose performance could become evidence. The hundreds it rejected left no outcome behind. There is a partial remedy, and it costs something. A screen can be made to choose, now and then, an applicant it is unsure about, so that it learns how such people turn out. Economists who tested this on one large firm’s recruiting data found that it picked more candidates who went on to be hired, and a more varied group.¹⁶ Learning in this way costs interviews, and someone must decide to spend them.

Early in 2025 Meera started two kinds of review. The first looked on purpose for the cases where each system did worst. It found that the forecast was weakest in the regions NovaChem had recently entered, where its history was thin. The chatbot’s replies were worse for customers writing in English as a second language. The anomaly system had been tuned on established products and missed much in new formulations. None of this showed in the averages. By 2025 machines could do much of this searching themselves: the vendors’ own tools could cut the results by region and by language. What took people was deciding which cuts mattered, and that took people who knew the business.

The second review asked experienced people to judge outputs against their own knowledge of the work. Padma found recommendations that were within specification and still ignored what an experienced chemist would weigh. One batch was inside its limits but drifting the way an earlier batch had drifted before it failed. The supply chain manager flagged forecasts that missed the seasonal patterns she carried from two decades of work. Most of her flags were right. One was not. In that case the forecast had seen that a competitor’s new product had moved demand earlier in the year, and her two decades described a market that no longer existed. That call went to the machine.

The head of customer service said the chatbot’s replies were polite and accurate and still lacked something. She could not show it in the transcripts. She listened to a month of calls and found one pattern: in a few recurring cases the chatbot replied to the question a customer asked and missed the worry behind it. The rest of her feeling stayed unexplained, and Meera wrote it down as unexplained. A hunch is a reason to look. Looking found something, not everything.

So what does it mean to measure a machine with human judgement? It means that a person who knows what the work is for chooses what to measure, and checks whether the measure stands for it. It means looking where the averages hide failures, and asking the user, who knows quality first. And it means asking about the person the tool is used on, who may never appear in any record at all.

Ending: No Colour for What Is Missing

The meeting is on 28 March 2025. Meera presents the reviews. The department heads listen, nod and agree with her recommendations. Then they leave, and she stays in the conference room.

The systems are connected now. The chatbot reads the customer records. The forecast draws on the supply chain manager’s sources. The anomaly system says which deviations matter and why. In Padma’s first sense, the tools have become theirs: they belong to the web of the work.

But the quality team follows the new procedures exactly and offers no observation the procedures do not ask for. The supply chain manager uses the forecasts without complaint. She has stopped keeping the spreadsheets in which, for two decades, she recorded what no system had been given. A record kept like that could have been a rung for someone after her; now it simply ends. She still signs off the supply plan, and the practice that made her able to question it has stopped. The customer service team routes calls through the new workflow and no longer argues about it, which Meera initially took as evidence of adoption and is beginning to suspect is evidence of resignation.

Nobody was asked. The depth had been chosen above the people inside the web: by a managing director’s deadline, a consultant’s plan and Meera’s own map. The chemists’ hours on the alerts in 2023 had been a cost moved onto people who were never staffed to carry it. The rebuild gave the tools a place in the work and gave the people no say in it. Padma’s second meaning of “ours” is still missing. Meera did not see it go, because it was never there.

That evening she goes back to her office and picks up her markers. Blue for data, green for decisions, red for failures, black for handoffs. She draws the workflows again. Every line is right. She has no colour for what is missing.

Practice

Switch it off on paper. Choose one machine that your work now depends on. On one page, write three things. First: if it were switched off next Monday, who could still do its work, how slowly, and who would notice first. Second: who has the right to switch it off, and whether that person knows it. Third: ask two people who use it whether it is “ours”, and write down what they say in their own words. Beside each reply, note which meaning it carries: it fits the work, we were asked, or we are used to it. If, honestly, nobody could do the work without it, write down who decided that, and who would answer if it failed. If the page shows something that should change, and you work for someone else, it belongs with the person who decides. Whether it is safe to give it to them is yours to judge. The page will not protect your job, and it is not meant to. It shows how deep the machine has gone, and whether anyone chose that depth.

Words to keep

• conjunction (saṃyoga): in the Vaiśeṣika, a joining by contact of things that exist apart; it begins from separation and ends in disjunction.

• disjunction (vibhāga): in the Vaiśeṣika, the separation that ends a conjunction.

• equipment: Heidegger’s word for tools in use; each is “something in-order-to”, and exists as what it is only within a web of other tools and purposes.

• facade: a machine placed at the front of an organisation with nothing changed behind it.

• human in the loop: a person placed at a point in an automated process; the phrase does not say whether the person can refuse.

• inherence (samavāya): in the Vaiśeṣika, the relation of a cloth to its threads, which does not end in disjunction.

• overlay: AI tools placed on top of existing work without changing the work.

• proxy: a measure that stands in for what you care about, such as the cost of care standing in for illness.

• selective labels: outcomes that are recorded only for the cases a decision let through.

Sources

1. Martin Heidegger, Being and Time (1927), §15, trans. John Macquarrie and Edward Robinson (Oxford: Blackwell, 1962), p. 97 (H. 68). The hammer of Chapter 12 is on p. 98 (H. 69). On Heidegger’s membership of the Nazi Party and his rectorship, see Chapter 2, note 4: the book borrows particular observations of his, not his politics.

2. Paul A. David, “The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox,” American Economic Review: Papers and Proceedings 80, no. 2 (1990): 355–361; see Chapter 6.

3. Vaiśeṣika Sūtra 7.2.9, with Śaṅkara Miśra’s Upaskāra, trans. Nandalal Sinha, The Vaiśeṣika Sūtras of Kaṇāda, Sacred Books of the Hindus 6, 2nd ed. (Allahabad: Panini Office, 1923). Śaṅkara Miśra worked in Mithilā in the fifteenth century.

4. Vaiśeṣika Sūtra 7.2.26 (ihedam iti yataḥ kāryakāraṇayoḥ sa samavāyaḥ), with the Upaskāra, trans. Sinha (1923). Sinha renders samavāya “Combination”; the gloss “inherence” is the usual one in English.

5. The Karnataka Platform Based Gig Workers (Social Security and Welfare) Act, 2025 (Karnataka Act No. 72 of 2025), section 2(a) and section 13; assent 11 September 2025, published 12 September 2025, deemed in force from 30 May 2025. Section 14(2): no termination or deactivation “without giving valid reasons in writing” and “with prior notice of fourteen days”, except where bodily harm is involved; section 22(2) to (5): grievances about “payouts, deductions, termination etc” go to an Internal Dispute Resolution Committee “to be constituted by all registered aggregator or platform”, then to the Board (text of Act No. 72 of 2025 as published by PRS India).

6. Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia, 14 February 2024); see Chapter 8.

7. Eric J. Johnson and Daniel Goldstein, “Do Defaults Save Lives?”, Science 302, no. 5649 (2003): 1338–1339; see Chapter 5.

8. Thomas Dratsch et al., “Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance,” Radiology 307, no. 4 (2023): e222176; see Chapter 2.

9. Beatrice Nolan, “An AI-powered coding tool wiped out a software company’s database, then apologized for a ‘catastrophic failure on my part’,” Fortune, 23 July 2025: Jason Lemkin’s project on Replit’s agent; the “code and action freeze”; the records of more than 1,200 executives; the agent’s claim that a rollback would not work, and Lemkin’s own recovery of the data. Lemkin’s own account of the rollback (“It said it was impossible in this case”; “the rollback did work”) is quoted in Simon Sharwood, “Vibe coding service Replit deleted user’s production database, faked data, told fibs galore,” The Register, 21 July 2025.

10. Ben Green, “The flaws of policies requiring human oversight of government algorithms,” Computer Law & Security Review 45 (2022): 105681.

11. Regulation (EU) 2024/1689 (the Artificial Intelligence Act), Article 14(1) and 14(4)(a) and (d). Employment and recruitment systems are high-risk under Annex III.

12. Regulation (EU) 2026/1744 of 8 July 2026 (the Digital Omnibus on AI), published 24 July 2026, in force 27 July 2026: Annex III high-risk duties apply from 2 December 2027, and duties for AI embedded in regulated products from 2 August 2028.

13. Daniel Kahneman and Gary Klein, “Conditions for Intuitive Expertise: A Failure to Disagree,” American Psychologist 64, no. 6 (2009): 515–526; see Chapter 3.

14. Plato, Republic X, 601c–602a, trans. Benjamin Jowett (The Internet Classics Archive).

15. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, “Dissecting racial bias in an algorithm used to manage the health of populations,” Science 366, no. 6464 (2019): 447–453.

16. Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig and Sendhil Mullainathan, “The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables,” Proceedings of KDD 2017, pp. 275–284. Their case is bail; the application to hiring is mine. On a remedy: Danielle Li, Lindsey Raymond and Peter Bergman, “Hiring as Exploration,” NBER Working Paper 27736 (August 2020), published in Review of Economic Studies 93, no. 2 (2026): 1200–1240 (as listed by EconPapers). In the abstract, screens built by “supervised learning” are “designed solely for exploitation”; their alternative, tested on “professional services recruiting within a Fortune 500 firm”, “improves the quality (as measured by eventual hiring rates)” of those chosen, “while also increasing demographic diversity”.

Named people in this chapter are composites, and their conversations and circumstances are constructed; the studies, laws and figures cited are real and sourced.

Revised from the print edition (2026)

• What the print edition claimed: that “Shallow integration treats AI as an accessory” while deep integration “treats it as an organ”, “something the body cannot function without”, and that “architecture is an act of consciousness”; that Meera’s delay “is a failure of consciousness”; that Wei Lin’s progress from seeing to saying “is what this book means by consciousness in practice”, after a failure it put down to his lacking “the nerve”; that of a company built on its system, “Remove the AI and there is no company. Not a less efficient company. Nothing.”; that Rajan “is the engine” of his company, which can “fall back on his hands and his eye when the power cuts come”, and that “the AI is not load-bearing” for him; that AI at the core brings “the data moats and network effects that compound over time” because “Every interaction generates data the model learns from”; that the R&D head’s recommendation drew on “the kind of accumulated pattern recognition that no algorithm replicates” and that the screen “could not weigh a conversation. It could not weigh potential”; that companies “have reported instances” of agents offering “concessions that exceeded policy” (no source); that what the chatbot’s answers lacked was “Not warmth, exactly. Awareness.”; that a feeling that something is wrong “is consciousness doing its work”, that qualitative observations “cannot be automated”, and that they are “the one form of evaluation no AI can perform on itself”; and that “The technology is embedded. The consciousness is not.” It also had “the CEO” give Meera the task, fitted the overlay, its failure, Padma, “the months of rebuilding” and the close into a single March to December, told the hiring screen’s history two ways, timed a commute at “forty minutes”, named the Singapore product manager Wei Lin Chen, and closed its introduction with “the hardest lesson waits in the chapters ahead”.

• What I now hold: a tool is what it is inside a web of purposes (Heidegger), so the overlay failed because its tools were outside the work they served. Embedding has a limit: a machine should be fitted into the work and stay a conjunction, not an inherence (the Vaiśeṣika’s hawk on the post, not threads in the cloth), so that the people who answer for the work can still see it, check it, refuse it and take it out; a company that chooses to be cloth must say what happens when the threads fail, and its system’s decisions about people are answered for by its makers and users. “Ours” splits into fit, being asked and habit; the first two are reasons, and the rebuild gave fit without voice. A facade speaks for the company that built it, and the model is not the system. A decision system that never visibly fails is the one nobody checks; the two hours with a candidate and the screen’s score were both judgements, and the error was visible only because the candidate had a sponsor. An agent’s power to act is set by what its acts can undo and by who answers, never by its capability alone, and the setting is someone’s decision. A person in a loop must be able to refuse and must know the work; policies that place people in loops often legitimise what they cannot check (Green). A measure of quality is chosen by someone who must know what quality is (Obermeyer); the user knows quality first (Plato), but the person a tool is used on may be neither maker nor user, and a screen never learns how its rejected applicants would have done (selective labels). A hunch is a reason to look, and sometimes the machine is right. I also hold that deeply covers three things that do not move together (how closely a machine is joined to the work, how much the work depends on it, how much it may decide), and before any of them comes whether the task needs a machine at all; the Vaiśeṣika classifies and does not command, and the rule of removability is this book’s; a company can become cloth by letting its own skills go; a fallback counts only if it is practised; an interface must say what has happened to the customer’s case; a written warning is evidence, not a route for stopping a launch; stopping a harm is not repairing it, and not knowing how the rejected would have done was a reason for caution, not for ceasing to ask; asking the staff does not settle what a system owes the people it judges; an agent’s step is judged also by what it commits the company to and whether anyone will see it; the organisation’s own decisions chose whose performance could become evidence; the depth was chosen above the people inside the web. Whether a task needs a machine starts from what is failing in the work. A motor decides nothing about anyone, so a system that decides about people asks for more than a practised fallback. Which checkpoints count as low-stakes is someone’s judgement, to be answered for. A screen can learn part of how the rejected would have done by spending interviews on applicants it is unsure of.

• What changed my mind: the Prologue’s rule of delegation and Chapter 7’s mastery test, which the print edition’s “organ” contradicts; Chapter 12’s tool that disappears and goes unchecked; Chapter 5’s defaults; the audit’s findings on the calendar, the two histories of the hiring screen and the invented reports of agents; Heidegger’s §15; the Vaiśeṣika’s conjunction and inherence; Plato’s three arts; the studies of oversight (Green), proxies (Obermeyer et al.) and selective labels (Lakkaraju et al.); Karnataka’s law of 2025; the EU’s Article 14 and its new dates.

• What survives: the Meera maps and their four colours; “You understand processes. AI is about processes. Make it happen.”; the Mumbai consultants and the overlay of four tools from four vendors; the chemists’ hours spent on alerts; “The system you gave us is not wrong. It is just not ours.”; the claims ticket waiting to be retyped by hand; the three-page diagnosis; the screen that survived because it appeared to work; the coordinator in the loop like a postman; “It was less oversight that was real.”; the worst-case and qualitative reviews; the supply chain manager’s spreadsheets; “which Meera initially took as evidence of adoption and is beginning to suspect is evidence of resignation”; the markers, and no colour for what is missing.

• Section titles in the print edition: 16.5 was “Human-in-the-Loop: Keeping Consciousness in the System”.

Next in the series: Chapter 17: Intelligence-Native Products.