What is Model Drift?

Every deployed model was validated against the world as it stood on a particular day. The world declines to stay that way. Model drift is what happens in the gap: the slow, silent divergence between what a model was proven to do and what it is now doing.

The model you validated is not the model you are running

Model drift is the change in a model’s behaviour over time, not because anyone changed the code, but because the world underneath it moved. The weights may be byte-for-byte identical to the day of sign-off; behaviour is not a property of the weights alone. It is a property of the weights meeting the data of the day, and when that data changes character, the same model gives different answers, delivered with exactly the same confidence as before. That is what makes drift dangerous: there is no error message. The system does not announce that it has quietly become a different system. It keeps producing fluent, plausible outputs. They are simply less true.

This is also why yesterday’s validation does not cover today’s model. Validation is a measurement, and like every measurement it is stamped with a date. It says: this model, against this data, at that time, performed like this. It is a photograph, and organisations habitually treat it as a permanent property. The accuracy from launch gets quoted in reviews long afterwards, describing an ancestor of the system actually in production. A model is validated at a moment and trusted for a duration, and drift lives in the gap between those two tenses. Nothing about a good launch validation is evidence about the model’s behaviour this quarter. It has the same status as an old measurement in any other domain: honest when taken, expired since.

Data drift and concept drift

Two conceptually distinct forces sit under the one word. Data drift means the inputs change: the population the model now sees no longer resembles the population it learnt from. A screening model trained on one applicant mix meets a different mix after the firm enters a new market. A demand model trained on one channel mix meets customers arriving through channels that barely existed at training time. The model is being asked questions from outside its experience.

Concept drift is subtler: the inputs can look the same while the meaning changes. The relationship between what the model sees and the thing it is predicting shifts. What predicted a reliable customer before a change in economic conditions may predict something else after it. What made a support ticket urgent last year may not make it urgent now. The model’s experience still covers the questions. It no longer covers the truth.

The distinction matters because the two are caught differently. Data drift can often be seen by watching the inputs. Concept drift frequently cannot: the inputs stay in range, the outputs stay confident, and the only witness is the outcome, discovered later.

The monitoring duty, and who owns it

Because drift is silent, monitoring is not a courtesy. It is the other half of deploying a model at all, and it must be owned by name. “The vendor monitors it” fails because the vendor cannot see your outcomes. “Everyone watches the dashboards” fails because a duty shared by everyone is owned by no one. Whoever owns it needs three feeds: the inputs (is the population shifting), the outputs (is behaviour changing shape), and, decisively, the scored outcomes (did what the model claimed come true). The first two are commonly instrumented. The third is the one most organisations lack, because it requires remembering what the model claimed and comparing it with what happened, which is a discipline of record-keeping, not of modelling. For high-risk systems the EU AI Act points in the same direction, expecting monitoring across the life of the system rather than proof at a point in time. (Educational context, not legal advice.)

One concrete example

Clearly illustrative, with no customer implied. A services firm forecasts weekly demand for a client operation using a model trained on several years of history. The client then changes how its product is sold, and the mix of customers arriving shifts in ways the history never contained. The inputs still look normal: volumes within familiar ranges, seasonality where expected. But the relationship between early-week signals and end-of-week demand has changed, and the forecasts begin to run persistently optimistic. Nothing alerts. Schedules are built on the forecasts, and service quietly degrades. The drift is finally noticed not by inspecting the model but by a review that compares its forecasts against what actually arrived and finds the errors trending one way. Concept drift, visible only in outcomes.

Drift and decision intelligence

You can only see drift if you remember what the model claimed and score what came true, which makes drift monitoring a special case of decision intelligence. In ONX, a model’s output is treated as what it is on the evidence hierarchy: modelled evidence, distinct from measured fact, with confidence weighted accordingly. Decisions taken on that evidence are scored afterwards against what was recommended and what was decided, and the outcomes ledger holds the comparison; a learning loop mines those scored outcomes into patterns only once enough evidence accumulates. That is drift monitoring by construction: when a model’s claims stop coming true, the record shows it, decision by decision, before the aggregate dashboards do. The duty still needs a named owner, and a human in the loop still makes the consequential calls. But the owner needs a memory, and that is what the system is for.