The Cost of Never Having to Know

I

The standard answer to autonomous systems is to keep a human in the loop. It sounds reassuring because it preserves the appearance of sovereignty; the machine may observe, infer, decide and act, but somewhere behind it stands a person who can still say no.

The assumption is that this person remains fully capable while the machine grows more capable around them. That is unlikely. Human competence is not preserved by title, authority or architecture. It survives through use.

II

Lisanne Bainbridge described the contradiction in Ironies of Automation in 1983. Engineers automate tasks because humans are unreliable, but whatever cannot be automated remains with the human. Over time, routine work disappears into the machine while ambiguity, novelty and failure remain outside it.

The machine takes the repetition. The human inherits the remainder.

This matters because expertise is not simply knowledge retained in the head. It is a relationship with a system built through repeated contact; seeing normal states, noticing slight deviations, intervening, being wrong, correcting, remembering. Judgment is accumulated almost invisibly through ordinary encounters. Remove those encounters and what remains may still look like knowledge, but it is thinner.

The crisis eventually demands what routine work used to produce.

III

Early automation conceals this problem well because it lives on inherited competence. The senior engineer who intervenes when an agent fails may understand the underlying system because they spent years operating it before the agent existed. Their judgment was formed in a world the automation is now removing.

For a while, this creates the impression that nothing has been lost. The machine performs the ordinary work and the experienced human remains available for the exceptional case.

But inherited competence has a generational limit. The engineer who enters later may learn the agent rather than the system beneath it. They learn how to supervise remediation, inspect summaries and approve actions without passing through the same sequence of mistakes, interventions and consequences that produced the judgment of the generation before them.

An organisation can retain humans everywhere and still lose the human knowledge on which the architecture quietly depends.

IV

Agentic AI makes the structure more dangerous because it automates not only action but interpretation.

The system collects evidence, decides what matters, constructs a hypothesis, acts on that hypothesis and then explains what happened. The human who arrives later does not encounter the event directly. They encounter an account of the event produced by the same system whose judgment is now under question.

Oversight begins to fold back on itself.

The operator appears to supervise the machine, but increasingly sees the world through the machine. Formal authority remains intact while epistemic independence weakens. The person can still reject the recommendation, but their reasons for accepting or rejecting it may already have been framed by the recommendation itself.

The machine no longer merely acts on reality. It mediates reality for the person meant to constrain it.

V

There is another problem hidden inside improving performance.

As automation becomes more capable, fewer cases reach humans. But the cases that remain are not a neutral sample. The ordinary incidents disappear first. Familiar faults, routine recoveries and obvious diagnoses are absorbed by the system. What remains is increasingly strange; incomplete evidence, conflicting signals, novel states, high consequence.

The human is therefore asked to practise less and perform at a higher level when finally summoned.

This is an unusual bargain. We remove the repetitions through which judgment is formed, then preserve the human as insurance against situations that require judgment.

The better the automation becomes, the more exceptional the human’s remaining work becomes. The more exceptional the work becomes, the less often the human encounters it.

Success slowly destroys the rehearsal for failure.

VI

Automation can also erase the signs from which understanding grows.

A system that restarts services, reroutes traffic, clears queues, adjusts capacity and repairs configuration may produce admirable reliability. Yet each successful intervention can also remove evidence of the instability underneath. Symptoms disappear before becoming stories. Incidents vanish before turning into institutional memory.

The system appears healthier because fewer humans experience its sickness.

Eventually something arrives that cannot be healed automatically. At that point the organisation discovers that it has not merely lost visibility into the current failure. It may have lost years of accumulated contact with the conditions that produced it.

A self-healing system can become a system that teaches its operators less about itself.

VII

The usual discussion treats human competence as if it were a fixed resource. It is not. It decays when unused, narrows when mediated, and disappears when a generation no longer has reason to acquire it.

This creates a feedback loop more important than the familiar story of machines replacing labour. As the system becomes more capable, human exposure falls. As exposure falls, independent competence becomes harder to sustain. As competence weakens, reliance on the system deepens. Greater reliance makes withdrawal still less plausible.

Dependency can therefore increase precisely because the automation works.

The most dangerous stage may not be when the machine is unreliable, but when it becomes reliable enough that understanding it no longer seems necessary.

VIII

This is why “human in the loop” may become one of those phrases that survives long after the reality it once described has disappeared.

The question is not whether a person remains somewhere in the architecture. The question is what kind of person remains there. Do they possess enough contact with the underlying system to form an independent judgment? Can they reconstruct events without accepting the machine’s explanation of them? Have they accumulated the experience required to recognize when the machine is wrong in a way the machine itself cannot articulate?

A human who cannot answer independently is not meaningfully in the loop. They are adjacent to it.

The distinction will become harder to see because the interface will still show the same controls. Approve. Reject. Escalate. Override.

Authority can remain visible long after capability has moved elsewhere.

IX

The deeper risk of agentic AI is therefore not that humans disappear from work. It is that they remain present after the conditions that made their presence valuable have been removed.

They will still carry responsibility. They may still sign off decisions, take escalations, answer for failures and appear on governance diagrams. But responsibility without sustained competence is only a slower form of dependence.

For a time, the system may continue to function on knowledge accumulated before autonomy became normal. After that, the inheritance begins to thin.

The machine keeps getting better.

The human stays in the loop.

And somewhere between those two facts, the loop stops being human.

Recommended reading

Ironies of Automation (1983) by Lisanne Bainbridge