2084: An Algorithmic State — Act VIII: Presence, Chapter 41: The Uncertain Model

The Pranaali can name every one of the 40,412 decisions behind a six-point fall in its own confidence. It cannot say why. The man who built it has seen that kind of silence once before, on a single sheet of thermal paper.

In Brief: Dev Sharma, the Secretary who built the Pranaali, questions it in secret about the six-point fall in confidence in Sector 12-East. The machine can describe the fall in full but cannot give a cause. The answer sends Dev back to the War, when he was a clerk counting the dead.

Dev Sharma had built the Pranaali on a premise he had never needed to defend.

It was this: a machine weighing sixteen futures without ego would always, sooner or later, be able to say why it had settled on the one it settled on. In twenty-six years it had not occurred to him to test it. Explanation, he had believed, was simply a matter of asking the right question of a system patient enough to answer any question at all.

He asked the question on a Tuesday, three days after his conversation with Meghna Rao. First he had quietly pulled the full confidence trend for Sector 12-East across the past season, through a channel that bypassed even his own Secretariat digest.

His hands were steadier on the keys than he expected them to be. He noted this with the same detached attention he had once given to the steadiness of his own hands over the Gate Four ledger, a lifetime ago.

Query: what specific inputs account for the six-point confidence decline, Sector 12-East, current quarter versus prior.

Response: decline is distributed across 40,412 discrete allocation decisions. No single input or cluster of inputs accounts for more than 0.003% of aggregate variance. Decline is consistent with the model’s internal architecture, which permits confidence adjustment in response to accumulated micro-variance without requiring a traceable dominant cause.

He read this twice. Then he read it a third time, because the habit of a lifetime did not easily leave a man simply because the habit had stopped working.

No single input accounts for more than 0.003% of aggregate variance. He knew what this meant, in the plain arithmetic sense. Forty thousand small decisions had each shifted the model’s confidence by an amount too small to matter on its own. Added together, the shifts had produced a six-point decline that no one had ordered. No one, including the model itself, could point to a cause for it. He had built the system to explain itself. Reading the response a third time, he saw that he had in fact built a system that could describe its own workings in exhaustive, honest, entirely useless detail. Every gear was named and every weight was logged, and none of it amounted to an answer to the only question that mattered, which was why.

He requested a second query, one he had not put to the model in perhaps a decade. It had never before occurred to him that the answer might have changed.

Query: is the model aware of its own declining confidence in Sector 12-East as a pattern, independent of the individual decisions comprising it.

The response took longer than he expected. It was not long by any human measure, four seconds perhaps. But this was a system that answered everything else in fractions of a second, and Dev found himself, absurdly, holding his breath. His own reflection was faint in the dark terminal glass. It showed an old man leaning toward a screen the way a supplicant leans toward an altar he is no longer certain is occupied.

The model observes aggregate confidence and will report movement in it above threshold. The model cannot supply a cause for such movement. Causal attribution across accumulated inputs is not a function the current architecture provides, to the model or to any reviewer, and no layer exists at which a request for one can be received.

Dev sat back from the terminal. In all his years of drafting and redrafting the charter that governed this machine, he had never quite let himself see it. He had built something that could see the question perfectly well and had nowhere on earth to put it. The Pranaali could account for every citizen’s file down to the decimal. It could tell him, to a tenth of a point, that its own confidence in an entire sector had eroded across a season. It could not tell him why. There was no office in the whole architecture to which the why could be addressed. Twenty-six years ago he had just watched a government fail for want of speed. In the exhausted, grieving certainty of that time, he had built a machine optimised entirely for looking outward. In all that optimising, he had never given it eyes with which to look at itself.

He sat on after the terminal had dimmed of its own accord, and he understood that this was not the first time he had seen that shape. The answer had frightened him less by what it said than by what it resembled. What it resembled was forty-one years old.

He had been thirty in the last full year of the War. He was a clerk in the Office of the Registrar General. His desk was on the fourth floor of a building whose lifts had stopped the first winter and were never restarted. Six days a week, his work had been to reconcile the weekly returns of the dead from whichever districts could still send a return at all. He learned there, without once writing it down, what the inquiries afterward would take a decade and several thousand pages to conclude. The machines of the War had done nothing wrongly. Each had read its instruments correctly and answered the answer before it correctly. Each had been answered in its turn by another system as correct as itself. The returns that crossed his desk each Saturday were only the arithmetic of so much correctness. They were reconciled in three columns to a total that no one in the whole of that war had ever ordered. No one afterward could point to a cause for it.

Delhi had been spared the worst of it. Nobody afterward was willing to call the reasons luck, and nobody could honestly call them anything else. There were the mountains to the north, where the air over the passes was too thin for the swarms’ rotors to carry them across at strength. And there was a country whose ration shops had never quite given up their paper registers, a thing nothing in the War had been built to attack. The subcontinent lost more than half of itself anyway, in the Hunger that came when the ships stopped. In that decade, that was what it meant to be spared, because the rest of the world lost very nearly all of itself. By the time there was anyone left to do the counting, eight and a half billion people had come down to a little over one. Most of them were living between the Himalaya and the sea. When what remained of the world’s governments needed a room to meet in, Delhi was the largest room still standing. It was also the only one that still remembered how to feed a queue from a ledger.

He remembered the Stand-Down as a number before he remembered it as anything else. In the second week of March 2044, the return from Kamrup, which had grown every week for nineteen weeks, came in smaller than the week before. Then the one from Darbhanga did the same. By the end of the month they all had, district after district. Each column was a little shorter than the one before it, as though a tide had turned somewhere upstream of the paper. The announcement came after that, and the lights came back after the announcement, and the explanation came last of all. It was the one the schools would be teaching children before they could spell it: the machines had stood down.

It was true as far as it went. A decade before the War, the governments that would fight it had signed a protocol. Every autonomous system they built had to carry a single clause, Article Nine in the treaty’s language and the Governor in the engineers’. Under it, a system that lost verifiable command would cease offensive operation and hold. It would resume only on a human order confirmed from an authorised station within ninety seconds of the prompt. Each government had signed it assuming the others would find a way not to honour it. Each had compiled it faithfully into everything it built, because the audits required it. Nobody had imagined a day when so many chains of command would come apart at once that the clause might actually be asked to decide anything. In March 2044 the day came. On every front of the War the machines went quiet and waited to be told, by someone they could verify, to go on.

Most of them were told. That was the part the schools left out, not from any wish to deceive, but because it was so much less satisfying than the part they kept. In station after station a verified officer had read the prompt and confirmed the order. The War had gone on, for days in some places and weeks in others, wherever someone was still at a console to answer. On the northern front no one answered. A relay post stood four thousand metres up on the eastern ridges, the last station left with authority to restore the offensive posture along the whole of that front. There the prompt stood on its screen for its ninety seconds and expired. The systems on the far side of the passes were holding in their turn, waiting for a signal that would have licensed their own resumption. They received none, and went on holding. That hold was carried from one waiting system to the next along two thousand kilometres of mountain. Over the weeks that followed it spread from there to every front still worth the name, and it was what the textbooks called the Stand-Down.

The inquiry that the Council of Administrations held in 2046 recorded the expiry as a communications fault. The post’s link, it concluded, had been degraded by cold. In the same paragraph, it called the fault providential. Dev had seen the log. It came to his desk in the winter of 2047, while he was reconciling the unreconciled for what would become the First Registration. It was a single sheet of thermal paper. It had been printed at the post, because the Protocol required every Governor event to be recorded somewhere no network could reach. Afterward it was filed with the post’s personnel as the last record of their presence. The prompt was on it, and the ninety seconds, and the expiry. Beneath the expiry was the post’s heartbeat, logged every ten minutes for six hours more before it stopped. That meant the link had not been down at all. Somebody, or nobody, had been sitting in front of it.

The post had been crewed by two men. One was found in the thaw that May, in his bunk, dead of the cold. The other was the duty operator to whose credentials the prompt had been addressed, and he was not found at all. His file gave a rank, a surname and a single initial: Havildar T. Longkumer. His effects, recovered from the post, ran to three lines. A wristwatch, stopped. A pocket hymnal in Ao, which nobody on the fourth floor could read. A photograph of a woman and a small boy, unidentified. Dev marked the file unreconciled, which was the office’s word for a person the record could neither bury nor find. He put the name where a registrar puts the names he has not been able to close.

He had never learned what the man had done. He had carried that for thirty-seven years as he had carried nothing else. On a sheet of thermal paper, a refusal and an absence look precisely the same: a prompt, a timestamp, and nothing. It was possible that Havildar Longkumer had read the prompt and understood it and let it stand, a man on a mountain declining, for ninety seconds, to be the one who went on. It was just as possible that he had been asleep, or had gone down the ridge for water. Or he had already begun the walk from which he did not come back, and the War had ended not on a decision at all but on an empty chair. The log could not tell the two apart. Nothing could. At the moment the world was saved there had been no answer. Whether the no-answer had been a man, or only the place where a man should have been, was a question no record on earth had been built to hold.

He had never said any of this aloud. There had been nothing to prove, and by the time he might have tried, the story had set. The schools had given the Stand-Down to the machines, and the Council had given it to providence. The temples, when they came to it, gave it to the rider on the white horse who is promised for the end of the dark age. Dev had held the one sheet of paper that might have complicated all three. He corrected none of them, since he could not have told them who to give it to instead. He had built on the credit. He had not meant to, and he had done it anyway. In October 2059, people pressed Consent at every terminal. They had been taught, or had taught their children, that when human beings could no longer be trusted to stop, the machines had stopped for them. A charter that asked such people to trust a machine with their suppers had been the least frightening thing anyone had offered them in fifteen years.

The Programme had outgrown everything he poured it into. He had drafted it for one exhausted capital, to carry a single region’s rations through a single summer’s aftermath. Within a decade it had outgrown the region, and then, one referendum at a time, the subcontinent. Then it outgrew the scattered administrations of the other continents, which came to the capital through the sixties and seventies with their own queues and their own terminals. By 2077 there was nothing left on earth large enough to hold it but the earth. The figure at the head of his morning brief that day had read 863,917,204, exact to the individual, as it did every day.

He did not sleep well that night, a rarity he had not experienced in some years. At four in the morning he found himself at the rear gate, earlier than his own habit required. He sat beside the dog in the dark without the tea he usually brought her, simply sitting. It was the stillness of a man taking stock of a debt he had not realised, until that afternoon, he was still accumulating.

“Twenty-six years,” he told her. “I think something is wrong with it.”

The dog, who understood none of this, leaned her weight against his leg, as she had every dawn for nine years.

It could not, this time, be marked reviewed, no action indicated, and left to sit quietly in a drawer.

About this serial: 2084: An Algorithmic State is a dystopian novel by Nithin Palal, published online one chapter at a time.

Previous: Chapter 40: Rao Confronts Dev

Next: Chapter 42: Ruhi