The Blind Cockpit

It is a quarter to eleven and the managing partner of a mid-sized agency is still at her kitchen table, laptop open, deciding what to do about a client that has gone quiet. The renewal is six weeks out. The account has felt wrong for a while: fewer replies, a rescheduled quarterly review, a tone in the last email she cannot quite read. She is tired, and she does the thing that a year ago she would not have thought to do. She opens an AI assistant and asks it, in plain language, whether she should push for renewal, offer a discount, or let the account go.

The answer arrives in seconds, and it is good. It is structured. It lays out three options with pros and cons for each. It offers a framework, "value at risk versus cost to serve," that she has to admit is rather elegant. It recommends a path: a proactive renewal conversation, framed around a modest scope reduction, to protect the relationship while defending margin. It even drafts the opening line of the email. It reads like counsel from a sharp, unflappable adviser who has seen a hundred accounts like this one.

And then, somewhere around the second read, she notices what it has actually done. It does not know that this account has slipped its delivery dates twice in the last quarter, because nobody wrote that down anywhere it could reach. It does not know that the sponsor who championed the agency internally left in March, or that the true margin on the work went underwater in month two, or that her best strategist has quietly asked to be moved off it. It knows none of the things that matter, because none of the things that matter were ever put in front of it. It has written a beautiful essay about a client it has never met, in the voice of someone who has. And it sounded right.

The real question is not power

The temptation, reading a scene like that, is to conclude that the tool is not good enough yet, and that a better one would have known. That is the wrong lesson.

The machine was not underpowered. On its own terms it performed beautifully. If you had asked a management consultant to produce the same memo cold, with no access to the account, they could not have done it faster or, in some respects, better. The problem was never the horsepower. The problem was what the horsepower was standing on, which was nothing at all. It was reasoning fluently about a situation it could not see.

Artificial intelligence has handed every leader a co-pilot that is fast, articulate, tireless, and occasionally, confidently, wrong. It will produce a plan for a market it has never studied, a diagnosis of a client it has never served, a forecast for a pipeline it has never seen, all in the same even, competent register. And that co-pilot is being bolted, at speed, onto businesses that cannot see their own instruments: firms that could not tell you on a Tuesday which of their live engagements are slipping toward a loss, because the information exists only in scattered systems and three people's heads.

Put a brilliant autopilot in a cockpit whose instruments are blind, and you have not made the aircraft safer. You have made it faster at going the wrong way.

That is the image to hold for the rest of this chapter. A powerful autopilot, flying a plane that cannot read its own altitude.

What the machine is actually doing

Strip away the branding and the demos, and at its core a large language model generates text by predicting likely next tokens from the context it has been given. It has been trained on an enormous quantity of human text, and from that text it has learned, with real sophistication, what tends to follow what in what contexts. When you give it a prompt, it does not look up an answer. It generates one, one token at a time, by repeatedly asking a single question: given everything so far, what is the most plausible thing to say next?

The generative mechanism rewards text that is plausible, which is to say text that reads like the kind of thing a knowledgeable person would write. Training and safeguards can push it toward what is true, and modern systems are explicitly tuned to be more factual, to follow instructions, and to draw on tools and retrieved sources. But the mechanism underneath does not independently guarantee truth. Most of the time, for most questions, plausible and true point in the same direction, which is why the tools are genuinely useful and why they feel like magic. When they come apart, the machine has no independent way of knowing. It has only its sense of what sounds right.

A language model is probabilistic. Ask it the same question twice and you may get two different answers, both delivered with equal composure. And when its uncertainty is not surfaced or constrained, it may fill a gap in its knowledge with a plausible completion rather than make the absence of evidence obvious. It can be confidently wrong, producing a citation that does not exist, a statistic that was never gathered, a fact that is simply invented, in exactly the same tone it uses for things that are correct. The industry has a gentle word for this, hallucination, but the mechanism is not a glitch. It is the system doing precisely what it was built to do, which is to continue plausibly, whether or not the plausible continuation happens to be real.

And, most important for our purposes: a language model has no access to your private context unless you give it that access. It did not train on your margins, your delivery history, your pricing decisions, your client's temperament, or the reason you walked away from a similar deal two years ago. Unless you place them in front of the model, deliberately and in a form it can use, they do not exist for it. It will not tell you it is missing them. It will write around the hole, smoothly, as though the hole were not there.

None of this is a criticism of the technology. It is a description of it. The models are a genuine advance, and the fluency is real.

Fluency is not knowledge

Human beings use fluency as a proxy for competence, and we do it almost without noticing. When someone speaks in clear, confident, well-organised prose, we grant them authority. We assume that behind the smooth sentences sits a body of knowledge, because in our experience that is usually a safe assumption. The command of language is evidence of the command of the subject.

Machine-generated text breaks that link, quietly and completely. The model produces the fluency without the underlying knowledge, and it does so by default, on every subject, whether it knows anything real about your situation or not. You are reading the confident prose of an expert and receiving the actual knowledge of a stranger who has never seen your books.

This is why the memo at the kitchen table was so persuasive and so empty at once. Every signal a human uses to gauge whether to trust a source was pointing green. The structure was expert. The tone was measured. The reasoning hung together. The only thing missing was any connection to the actual client, and that absence produces no visible signal at all. A guess and a grounded judgement look identical on the page when both are written by a machine that is fluent in the language of judgement.

In the last chapter we said that trust is built by a consistent record of good decisions over time. These systems manufacture the appearance of a trustworthy source, the fluency and the confidence, without the substance that is supposed to earn it: the signal of good judgement, detached from the reality of it. A firm that learns to trust the signal, because the signal is so good, is being trained to misplace exactly the asset it can least afford to spend carelessly.

The answer machine and the reasoning partner

There are two fundamentally different ways to bring one of these systems into a business, and almost all of the confusion in the market comes from mistaking one for the other.

The first is AI as an answer machine. You ask a question, you receive a plausible answer, and the answer is built from the general patterns of human text plus whatever you happened to type into the box. This is the mode most people are using today, because it is the mode that requires nothing of the organisation. It is genuinely helpful for a wide class of tasks: drafting, summarising, explaining a concept, generating options, rephrasing a difficult email. For anything where plausible really is good enough, and there are many such things, the answer machine earns its keep.

But the answer machine has a hard ceiling, and the ceiling is the one we have already named. It cannot reason about your actual situation, because it cannot see it. Ask it a question about your specific client, your specific margin, your specific decision, and it will still give you a competent general answer, dressed convincingly as a specific one. It produces the same fluent plausibility whether you are asking it to suggest a subject line or asking it whether to fire a client, and it gives you no way to tell, from the output, how much it actually knew.

The second way is AI as a reasoning partner, and the difference is entirely a matter of what the machine is standing on. A reasoning partner is grounded in the firm's real decisions and outcomes: the actual state of the engagement, the history of what was promised and what was delivered, the record of similar situations and how they turned out, the evidence that a human adviser would insist on seeing before opening their mouth. When you ask a grounded system about the quiet client, it does not reach for the average of the internet. It reaches for this client. It can say that delivery slipped twice, that the margin went underwater in month two, that the last three accounts you handled this way were saved by a scope conversation and the two you discounted instead churned anyway. It reasons over your evidence, and, crucially, it can show you the evidence it reasoned over, so you can check its work.

That last property is the whole game. The reasoning partner can be audited. You can ask it not only what it concluded but what it stood on, and you can inspect whether the ground was solid. The answer machine may offer you a conclusion, and even an explanation of sorts, but not a verifiable chain from your evidence to its conclusion, because your evidence was never beneath it to begin with. One of these is a tool for producing content. The other is a tool for producing judgement. They look almost identical in a demo. They are not remotely the same thing, and the difference is the presence or absence of evidence.

Why AI without evidence can be worse than no AI

It is tempting to think that an ungrounded answer machine is at least better than nothing. A guess dressed as counsel is surely no worse than a guess, and the prose is nicer. This is the most dangerous piece of intuition in the whole subject, and it is wrong.

Consider what happens without the machine. Faced with the quiet client and no tool, the managing partner knows she is partly guessing. She feels the gaps in what she knows, and the feeling does useful work: it makes her cautious, it makes her pick up the phone to the strategist who is actually on the account, it makes her go and find the margin number before she commits. The discomfort of not knowing is not a bug in human decision-making. It is a safety mechanism. It is the friction that sends us looking for evidence before we act.

Now add the ungrounded machine. The guess comes back not as a guess but as a memo: structured, confident, framework-backed, ready to act on. The friction is gone. The very discomfort that would have driven her toward the evidence has been smoothed away by prose that feels like it already contains the evidence. She does not go and find the margin number, because the memo did not seem to need it. The machine has not filled the gap in her knowledge. It has concealed it, and concealed it more effectively than she could have concealed it from herself.

AI without evidence does not remove the guess. It launders it, converting a hunch into confident prose that a reasonable person will act on.

That is the precise harm. An ungrounded system takes an input everyone would treat with suspicion, a guess, an assumption, an average, and returns it wearing the clothes of analysis, stripped of every marker of its origin.

And it moves at a scale that matters. A single laundered guess at a kitchen table is one bad decision. But these tools are cheap and fast, and organisations are pouring them into every seam of the business: proposals, pricing, forecasts, client updates, board papers, staffing plans. Each becomes a place where an ungrounded system produces a confident artefact that someone downstream treats as grounded, because it is fluent and came from a tool everyone is told to trust. The plausible guesses compound, decisions get made on prose that was never connected to reality, and because nobody recorded what the decision rested on, nobody can later trace where it went wrong. No AI leaves you knowing you are in the dark. Ungrounded AI turns on a light that illuminates nothing and convinces you that you can see.

A powerful autopilot, blind instruments

A modern autopilot is an extraordinary piece of engineering. It can hold a heading, maintain an altitude, manage a descent, and fly a smoother, more precise path than most human pilots most of the time. But it does not perceive the world directly. It flies on the readings its instruments hand it: the altimeter for height, the airspeed indicator for speed, the attitude indicator for whether the aircraft is level or banking, climbing or diving. The autopilot is only ever as good as those instruments. It has no independent sense of altitude. It has the number the altimeter reports, and it acts on that number with total, mechanical confidence.

Which is why one of the most dangerous situations in aviation is not an engine failure or a storm, but an instrument that has quietly stopped telling the truth. A blocked sensor, a frozen probe, a source of pressure that has iced over, and the altimeter now reads a height the aircraft is not at, or the airspeed indicator reports a speed that is not real. The instruments do not announce their failure. They display a wrong number in exactly the same crisp, confident way they display a right one. And the autopilot, faithful and powerful, flies the wrong number with the same precision it would fly the right one, straight and level and calm, into the side of a mountain. The competence of the autopilot is not a defence. It is what makes the error lethal, because a competent system executes a wrong instruction thoroughly.

Now look at a business the same way. Its instruments are the readings a leader flies by: whether the work is profitable, whether the pipeline is real, whether delivery is level or nose-diving, whether a client is climbing or quietly losing altitude. In most firms those instruments are blind, or nearly so. The information that would tell you the true margin on the quiet account is scattered across a CRM, a time tracker, a finance system, and a delivery lead's memory, and no instrument in the cockpit assembles it into a reading you can trust. A later part of this book gives that condition its proper name and its full diagnosis. For now the point is only this: the instruments do not work, and everyone has learned to fly on feel.

Into that cockpit we are now installing a superb new autopilot, which is what bolting an ungrounded AI onto a business amounts to. It is more capable than the pilots, faster and tireless and unshakeably composed, and it is flying on the same blind instruments, filling the missing readings with plausible guesses. The upgrade did not fix the blindness. It removed the last of the hesitation that blindness used to produce.

What executives actually need from AI

Ask a leader what they want from these tools and the honest answer is not more content. Content is already the cheapest thing in the building. The proposals, the summaries, the first drafts, the restated emails: all of that is now nearly free, and its abundance is closer to a problem than a solution, because it buries the few things that matter under a rising tide of plausible material that all reads as though it were considered.

What an executive actually needs is scarce, and it is not text. It is grounded judgement, governed evidence, and auditable reasoning. Take the three apart, because each is doing real work, and because the first of them, the one everybody reaches for, is the one most easily misunderstood.

Grounded means the machine's reasoning stands on the firm's real decisions and outcomes, not on the average of the public internet and not on whatever happened to be typed into the box. When it tells you something about the quiet client, it is reasoning about this client, from this firm's actual record, or it tells you plainly that it cannot, rather than papering the gap with fluent prose. Grounding is the difference between an adviser who has read the file and one who is improvising in the voice of someone who has. But grounding is necessary, not sufficient. A system grounded in stale, partial, or contradictory records is standing on ground that will not hold.

Governed means the evidence itself is kept honest: current rather than out of date, attributable to a source you can name, relevant to the decision in front of you, and open about its own uncertainty. Internal data is not automatically good data. A firm's own records can be stale, incomplete, biased by how they were captured, or flatly contradictory, and a model reasoning confidently over bad evidence is only a more local version of the same failure. The answer is not to abandon grounding but to govern it, a discipline this book returns to at length, because grounding is worth having only once the evidence beneath it can be trusted.

Auditable means you can see the working. You can ask what the conclusion rested on, inspect the evidence, and check whether the ground was solid, the way you would interrogate any adviser whose recommendation carried weight. This is the property that ungrounded content can never have, because there is nothing underneath it to inspect, and it is the property that turns a machine from a producer of confident guesses into a partner whose judgement you can examine before you rely on it.

Trust is what these three produce together. The firms that understand this will stop asking AI to write more and start asking it to help them decide well, which is a different request that most of them are not yet equipped to make, because the evidence a reasoning partner would stand on is not yet captured anywhere it could reach.

But won't the models just solve this themselves?

Here is the objection that dissolves most conversations about AI before they reach anything useful. The objection runs: all of this is a temporary state of affairs. The models are improving at a remarkable pace. Whatever they get wrong today, a better version will get right tomorrow. The grounding problem, the hallucinations, the confident errors, all of it is just an artefact of an early technology, and waiting for the next generation is the rational strategy. Why build anything, when the tool will soon build it for you?

The answer is that this objection misdiagnoses what is missing, and the misdiagnosis matters because it points every investment in the wrong direction. What a better model improves is capability: fluency, reasoning, breadth, the sophistication of what it can produce from what it is given. What no model can improve, from the outside, is grounding: access to your decisions, your outcomes, your private context, the specific evidence of your specific firm. It was never in the training data and it never will be, because it lives inside your business, most of it unrecorded, and a model trained on the whole of human text still has precisely zero knowledge of the margin on your quiet client.

So consider what actually happens as the models get better while the grounding stays absent. The prose gets more fluent. The reasoning gets more sophisticated. The frameworks get more elegant. And the errors, when the machine is guessing about your firm, get more persuasive, because a more capable model produces a more convincing account of things it still does not know. The kitchen-table memo of the next generation will be better written, better structured, and exactly as blind to the actual client.

A better model with no access to your decisions and outcomes is a better guesser, not a wiser adviser. Capability without grounding scales confidence, not correctness.

The thing improving is not the thing that is broken. Waiting for a more powerful model to fix a grounding problem is waiting for a faster autopilot to fix a blind altimeter. The models will keep getting better. That is exactly why the grounding cannot wait, because every increase in capability, absent grounding, is an increase in the persuasiveness of the mistakes.

The pilot, not the autopilot

If the instruments are blind, the machine cannot save us, however capable it becomes, because its capability is not the thing in short supply. What is in short supply is a firm that can see itself: that captures its decisions and their outcomes, so that judgement, human or machine, has something true to stand on. Building that is the work of the rest of this book, and we will come back to the co-pilot in a later chapter, once it has instruments worth reading, and show what grounded machine judgement can actually do.

But first there is a prior question, and the aircraft has been pointing at it all along. We have spent this chapter on the autopilot: what it is, what it can and cannot see, why its confidence is not the same as competence. We have said almost nothing about the pilot. And if the instruments are blind and the autopilot flies on guesses, then the hardest and most human question is not how to improve the machine. It is what it means to fly the plane at all: to lead, to decide, to hold responsibility for a firm you cannot fully see, in an age when a very fluent voice in the cockpit is always ready to tell you, with total composure, exactly where it thinks you are. That is a question about leadership, and it is where we turn next.