Explainable AI reasoning — The test of an explanation is not whether it sounds plausible. It is whether the person who knows the process can disagree with a specific part of it.
Explainable AI reasoning: bu ne anlama geliyor
Explainability is usually demonstrated with feature importance: this decision was driven 40% by seasonality, 30% by price. Show that to a planner with fifteen years on the category and watch what happens.
Nothing happens. There is nothing to argue with. The explanation is about the model, not about the business.
What a usable explanation looks like
“Recommending 400 units rather than the usual 250 because the promotion starting on the 14th resembles the one in March, which lifted this line 62%, and current stock covers eleven days against a fourteen-day lead time.”
Now the planner can say: March is not comparable, because the competitor was out of stock. That objection is specific, checkable, and — crucially — is information the system did not have.
The compounding effect
An explanation that can be disputed generates corrections. Corrections are training data of a kind that no amount of history provides: they encode why an expert disagreed.
Feature importance generates nothing, because there is nothing to say back to it.
Why feature importance persists anyway
Because it is available. Every framework emits it, it requires no domain knowledge to produce, and it satisfies a procurement question — “is the model explainable?” — with a yes.
It is explainability aimed at the wrong audience. A data scientist debugging a model genuinely needs to know which features carry weight. A planner deciding whether to accept a recommendation needs to know what the system believed about their business, which is a different statement entirely.
Both get called explainability, and the second rarely gets built because the first ticks the box.
The components of an explanation someone can dispute
Four, and each has to be specific.
The comparison: what past case is this being reasoned from. Not “seasonality” but “the March promotion on this line”.
The quantity: what that case implied. Not “high demand” but “a 62% lift”.
The constraint: what bounded the answer. Not “inventory considerations” but “eleven days of cover against a fourteen-day lead time”.
The counterfactual: what would change the recommendation. Not offered by most systems at all, and the component experts find most useful, because it tells them exactly which of their objections would matter.
Corrections are a different kind of data
A history of outcomes tells a model what happened. A correction tells it why an expert believed something would not happen — and that belief usually rests on a fact the system does not hold.
“March is not comparable, the competitor was out of stock” adds a variable. The next model can carry competitor availability. No volume of sales history would have surfaced that, because the absence of the variable is invisible from inside the data.
This is why the disputable explanation compounds and the feature chart does not: one generates new inputs, the other generates agreement or silence.
What this requires of the system
That the reasoning be constructed rather than extracted. A model does not produce sentences about March; a layer above it selects the comparable cases, the binding constraint and the sensitivity, and states them.
That layer is engineering rather than modelling, and it is where most of the value of an explainable system actually sits.
The test before deployment
Show three recommendations to the person who will receive them. If they cannot produce a specific objection to any of them, the explanations are not working — an expert who has seen thousands of these decisions always has objections to some.
Silence is not agreement. It is the absence of anything to disagree with.
A note on the word itself
Explainability has become a compliance term, and compliance terms attract the cheapest implementation that satisfies them.
The useful version is not a regulatory artefact. It is the mechanism by which an organisation’s expertise reaches a system that would otherwise have to rediscover it from data it does not have.