Do Deeper Transformer Layers Actually Explain Better?

When we explain a transformer’s prediction, we usually look at its two ends. Attribution methods such as SHAP [4] or Integrated Gradients [5] assign a score to each input token for a given output. In between, the input is rewritten twelve times: each layer of BERT [10] produces a new representation of every token, and each one arranges the data differently in a 768-dimensional space.

That raises a natural question. If the geometry of these representations changes with depth, does it say anything about how explainable the model is at that depth? Suppose a layer packs its inputs onto a simple, low-dimensional surface. Are attributions computed there more stable, or more faithful, than at a layer where the data spreads across many directions? Or is it the other way round?

Our IEEE BigData 2025 paper [1] set out to test this with one geometric quantity, local intrinsic dimensionality. This post goes through what we measured, what reproduces, and where the evidence stops.

Why explanation quality is hard to pin down

“Explanation quality” covers at least two different properties. Plausibility asks whether an explanation looks reasonable to a person. Faithfulness asks whether it reflects what the model actually used. The two can come apart: attention weights, for instance, often look plausible while failing basic faithfulness tests [6]. Faithfulness is usually measured by intervention. Remove the tokens an explanation calls important and see whether the prediction changes (comprehensiveness), or keep only those tokens and see whether it survives (sufficiency) [7].

This matters for everything that follows. The quantity in our paper is the mean absolute SHAP value: how much attribution mass the method assigns, on average. That is a measure of attribution magnitude. A larger magnitude does not mean a more faithful explanation; it can just as well mean a less calibrated model or a different output scale.

What local intrinsic dimensionality measures

A set of points can live in a high-dimensional space while varying along only a few directions. BERT’s vectors have 768 coordinates, but the documents may differ from each other along far fewer independent directions. Intrinsic dimensionality tries to count those directions, and local intrinsic dimensionality counts them in the neighbourhood of each point.

The intuition comes from counting neighbours. In a \(d\)-dimensional neighbourhood, the number of points within distance \(r\) grows roughly like \(r^{d}\). Double the radius on a curve and you capture about twice as many points; double it on a plane and you capture about four times as many:

The maximum-likelihood estimator we used [2][3] turns this into a formula. For a point \(x\), take the distances \(r_1(x)\le\dots\le r_k(x)\) to its \(k\) nearest neighbours. Then

If the near neighbours are almost as far away as the \(k\)-th one, the log-ratios are close to zero and the estimate is large: space is “roomy” in many directions. If the near neighbours are much closer than the \(k\)-th, the estimate is small. A layer’s LID is the average of \(\widehat{\mathrm{LID}}(x)\) over all documents. We used \(k = 20\).

The experiment

We kept the model and data deliberately simple, so that any change in LID could be attributed to depth rather than to training.

One detail is worth knowing because it explains a gap in the results. At the embedding layer (layer 0), BERT’s [CLS] vector is the sum of the [CLS] token embedding, the position-0 embedding and the segment embedding. None of these depend on the document, so every document has exactly the same layer-0 [CLS] vector. All neighbour distances are zero and LID is undefined, so layer 0 is excluded.

This is the part of the study that stands on its own: no attribution method is involved, only the representations and a distance computation.

What we observed. LID rises from 14.53 at layer 1 to 22.78 at layer 12, an increase of 57%. The rise is not monotonic. After reaching 21.81 at layer 7, LID falls to 19.02 at layer 8 and needs four more layers to exceed its layer-7 value. Across the 12 layers, the Pearson correlation between LID and layer index is 0.772, and Spearman’s \(\rho\) is 0.818. Because the model is frozen and the estimator has no random component, rerunning the code reproduces these values exactly.

How we read it. As a document passes through BERT, its [CLS] vector moves into neighbourhoods that vary along more directions. That fits the broad picture from other work on intrinsic dimension in deep networks, where the dimension of representations changes systematically with depth rather than staying flat [8][9]. The drop at layer 8 is the most interesting feature of the curve. The paper suggests a transition from syntactic to semantic encoding, but we did not test that, and it remains a hypothesis.

This was the question in the paper’s title. The paper answers it with a correlation: it reports a mean SHAP score for each layer (its Table I) and a Pearson correlation of r = 0.7721 (p < 0.01) between LID and SHAP across the 12 layers, with Spearman \(\rho\) = 0.736. It also states that Integrated Gradients could not be computed.

Trying to reproduce this from the released materials, we found the following.

• The released code does not compute per-layer attributions. It computes one model-level SHAP summary. In the included run, SHAP did not complete and the script fell back to a confidence score. Integrated Gradients returned a placeholder value, consistent with the paper’s statement that IG could not be computed. The repository’s README documents both.
• The figure printed as the per-layer SHAP figure (Fig. 2) shows these run-level summary values (fallback SHAP, IG placeholder, LID mean, LID standard deviation), not SHAP by layer.
• The reported r = 0.7721 matches, to four decimals, the correlation between LID and layer index that the released code computes. Correlating LID with the SHAP column of Table I gives 0.918 instead.

How we read it. We cannot reproduce the per-layer SHAP values in Table I from released code or logs, and the correlation the paper reports as LID–SHAP coincides with the LID–depth correlation. Until the attribution analysis is re-run with code that others can execute, the link between LID and explanations should be treated as an open hypothesis, not a finding. The LID curve in Question 1 is unaffected.

Why a layer-wise correlation would not settle it anyway

Suppose the attribution values were fully reproducible. A strong correlation between LID and SHAP across layers would still be weak evidence, for three reasons.

1. Depth is a confounder. Both quantities are measured once per layer, and layers are ordered. Any two quantities that drift with depth will correlate across layers, whether or not one has anything to do with the other. LID already correlates with depth at 0.77. A meaningful test would ask whether LID predicts attribution behaviour beyond what depth predicts. For example, compare across models with different depth profiles, or control for layer index.
2. Twelve points are not twelve independent observations. Neighbouring layers share most of their computation through the residual stream. A p-value that treats the 12 layers as independent samples overstates the evidence.
3. Magnitude is not faithfulness. Even a perfect relationship between LID and mean |SHAP| would show that higher-dimensional layers receive more attribution mass, not that their explanations are more correct. In a model that was never fine-tuned, attributions explain a randomly initialised classification head. Their faithfulness to any meaningful decision is not defined.

This is the central conceptual point. Representational complexity and explanation faithfulness are different properties, and a geometric measure can only become a proxy for the second after it has been validated against intervention-based tests.

What a sound version of the test would look like

The original question is still worth asking. A version that could answer it would:

• fine-tune the model on the task, so that attributions explain an actual decision;
• compute attributions with respect to each layer’s representation (for example, layer-conditional Integrated Gradients), with fixed seeds and released code;
• score the explanations for faithfulness (comprehensiveness and sufficiency [7]), not only magnitude;
• compare several models and datasets, including a randomly initialised BERT as a control for architecture-driven geometry;
• test whether LID adds predictive value after controlling for depth;
• report the sensitivity of LID to \(k\), sample size and distance metric.

What we learned

1. BERT’s [CLS] geometry grows more complex with depth, but not smoothly. LID rises by 57% from layer 1 to layer 12, with a sharp, reproducible drop at layer 8.
2. The explanation side of the study is not yet supported by reproducible evidence. The released code does not produce per-layer attributions, and the reported correlation coincides with the LID–depth correlation.
3. Correlations across layers are confounded by depth. Anything that trends with depth will correlate with LID.
4. Complexity is not faithfulness. A geometric property of a representation can only stand in for explanation quality after it is validated against interventions.

Limitations and open questions

• One model, one dataset, one representation, one \(k\). Does the layer-8 drop appear in other BERT-family models, in fine-tuned models, or for token-level representations?
• What is the drop at layer 8? A change in anisotropy, a reorganisation of the residual stream, or an artefact of the estimator? Comparing several intrinsic-dimension estimators would help.
• Does LID carry information about faithfulness once depth is controlled for? This is the question the paper’s title asks, and it is still open.
• The paper’s methodology text and the released code differ in sampling, truncation and distance metric. The numbers here follow the code.

@inproceedings{roufas2025deeper,
  title     = {Do Deeper Layers Explain Better? An {LID}-Based Study
               of Transformer Explainability},
  author    = {Roufas, Nikolaos and Kanavos, Athanasios and
               Karamitsos, Ioannis and Al-Hussaeni, Khalil and
               Maragoudakis, Manolis},
  booktitle = {2025 IEEE International Conference on Big Data (BigData)},
  pages     = {6552--6559},
  year      = {2025},
  doi       = {10.1109/BigData66926.2025.11402387}
}