What is Explainability?

Every AI system can produce an answer. Far fewer can survive the question that follows: why do you believe that? Explainability is the property that separates the two, and it is routinely confused with two neighbours that look similar and guarantee much less.

What explainability means

An AI system is explainable when it can show its reasoning in terms a responsible human can inspect and challenge. Every word is doing work. Show, not merely possess: reasoning that exists but cannot be surfaced is no use to anyone accountable. In terms a responsible human can inspect: the audience is the person who answers for the decision, not the person who built the model, so the explanation must speak the language of the decision (which facts, which trade-offs, what would change the answer), not the language of gradients. And challenge: an explanation the human can only receive is a briefing. The point of an explanation is that it can be tested, argued with, and, when it fails the argument, overruled.

Explainability, interpretability, and post-hoc rationalisation

Three terms, often swapped, meaning three different things. Interpretability is a property of the model’s internals: a small decision tree or a simple scoring rule is interpretable because a person can read the mechanism itself and watch inputs become outputs. It is the strongest form and the least available: many capable models are not interpretable in this sense, and no amount of documentation makes them so.

Explainability is a property of the system around the model. Even where the internals are beyond inspection, the system can still give an account of its output: which evidence contributed, of what quality, which considerations ranked where, and what would have changed the conclusion. The account is inspectable and testable even when the mechanism is not.

Post-hoc rationalisation is the counterfeit. It is a fluent story generated after the output, plausible in form, with no guaranteed connection to what actually drove the answer. The test that separates it from explanation is falsifiability: would this account have been different if the answer had been different? A real explanation constrains the conclusion; change the evidence and the conclusion should change. A rationalisation is compatible with any conclusion; it would have justified the opposite answer just as smoothly. Systems that generate confident narratives are abundant. Systems whose narratives can fail are rare.

The practical bar: why do you believe that?

Definitions of explainability sprawl, so it helps to compress the subject into one operational question. Can the system answer “why do you believe that?”, and is the answer evidence rather than eloquence? The practical bar for explainability is a single question, why do you believe that, answered with evidence a responsible human can inspect, challenge and overrule. Evidence means the actual facts behind the conclusion, each with its quality visible: measured, modelled, inferred, stated or unmeasured, in the terms of the evidence hierarchy. An answer built on measured facts should look different, to the person reviewing it, from an answer built on someone’s stated intentions. If every answer looks equally confident regardless of what it rests on, the system is not explaining. It is performing.

One concrete example

Clearly illustrative, with no customer implied. Two systems each recommend delaying a programme go-live. The first presents the recommendation with a risk rating and, when questioned, restates the rating in longer sentences. There is nothing to grab: no fact to check, no assumption to contest, no way to establish whether the caution is grounded or reflexive. The reviewing director can accept it or ignore it, which means the system’s intelligence is unusable precisely when it matters most. The second system shows its grounds: the facts that drove the recommendation, each labelled by quality, including one load-bearing claim that turns out to be merely stated, a delivery date asserted in a meeting and never confirmed. The director challenges that fact, a confirmed date replaces it, and the recommendation updates. That is explainability observed in the wild: not the elegance of the account, but the fact that challenging it changed something.

Explainability and decision intelligence

Explainability is often framed as an AI feature. It is better understood as a governance requirement that happens to fall on AI, and within decision intelligence it is structural rather than optional. ONX keeps its AI explainable and challengeable as an operating stance, and applies it most strictly where the stakes are highest: candidate matching in Hiring is treated as high-risk under the strict reading of the EU AI Act, with human review gates, so the human at the gate has something real to review. Cross-domain compound patterns are held to the same standard by design: every detected pattern is explainable, challengeable, traceable and falsifiable. And because a human in the loop always makes the consequential call, with overrides recorded (who, when, against what evidence), explanation and oversight close into a loop: the system shows why it believes what it believes, and the human is remembered for what they did about it.