Concepts & Methods
This page provides in-depth explanations of the key mechanisms and explainability methods used throughout this report.
1 Label Attention
1.1 Motivation
In a standard FastText or FATE model, once token embeddings are computed the sequence must be collapsed into a single fixed-size vector before the classification head. The default strategy is masked mean pooling: sum the embeddings of all real tokens and divide by their count.
\[ \mathbf{s} = \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \mathbf{e}_t \in \mathbb{R}^{d} \]
where \(\mathcal{T}\) is the set of non-padding positions and \(\mathbf{e}_t \in \mathbb{R}^d\) is the embedding of token \(t\). This sentence vector \(\mathbf{s}\) is then mapped to \(n_\text{classes}\) logits by a single linear layer \(W \in \mathbb{R}^{n_\text{classes} \times d}\):
\[ \hat{y}_c = W_c \cdot \mathbf{s} + b_c \quad \forall\, c \in \{1, \ldots, n_\text{classes}\} \]
The key limitation: all classes share the same sentence representation. For a text like “fast delivery but the screen cracked after a week”, the word fast is highly relevant for a 5-star prediction but irrelevant for a 1-star one, yet both classes look at the same pooled vector.
Label Attention replaces mean pooling with a cross-attention step that produces a different sentence representation for each class, by letting each class attend selectively to the tokens most relevant to it.
1.2 Mechanism
Each class \(c\) is assigned a learned embedding \(\ell_c \in \mathbb{R}^d\). These embeddings are collected into a matrix \(L \in \mathbb{R}^{n_\text{classes} \times d}\) that is trained jointly with the rest of the model.
The mechanism is a standard multi-head cross-attention where the label embeddings act as queries and the token embeddings act as both keys and values:
\[ Q = L \, W_Q, \quad K = X \, W_K, \quad V = X \, W_V \]
with \(X \in \mathbb{R}^{T \times d}\) the token embedding matrix (\(T\) = sequence length), and projection matrices \(W_Q, W_K, W_V \in \mathbb{R}^{d \times d}\). For \(n_h\) heads with \(d_h = d / n_h\):
\[ \text{head}_i = \text{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{d_h}}\right) V_i \quad \in \mathbb{R}^{n_\text{classes} \times d_h} \]
Padding positions are masked to \(-\infty\) before the softmax so they receive zero attention weight. The heads are concatenated and projected back:
\[ \text{LabelAttn}(L, X) = \left[\text{head}_1 \| \cdots \| \text{head}_{n_h}\right] W_O \quad \in \mathbb{R}^{n_\text{classes} \times d} \]
Each row \(c\) of this output is a class-specific sentence representation \(\mathbf{s}_c \in \mathbb{R}^d\), a weighted summary of the tokens that class \(c\) found most relevant. The classification head then applies a shared linear projection \(W_\text{head} \in \mathbb{R}^{d \times 1}\) independently to each row:
\[ \hat{y}_c = W_\text{head} \cdot \mathbf{s}_c + b \quad \in \mathbb{R} \]
1.3 Interpretability angle
The attention weight matrix \(A \in \mathbb{R}^{n_\text{classes} \times T}\) is a natural by-product of Label Attention: row \(c\) tells us, for the predicted class \(c^*\), how much each token contributed to the class-specific representation. Unlike post-hoc explanation methods, these weights are intrinsic: they are directly computed by the model at inference time, with no additional forward passes.
Attention weights as explanations: a word of caution. The debate on whether attention weights constitute faithful explanations is live in the literature. Weights quantify how much each token was used to build the representation, but the downstream linear layer may assign different importance to the resulting dimensions. The weights are nonetheless a useful lightweight signal, especially when combined with gradient-based methods.
2 Integrated Gradients
2.1 The attribution problem
Given a trained classifier \(F: \mathbb{R}^d \to \mathbb{R}^{n_\text{classes}}\) and an input \(\mathbf{x}\), feature attribution asks: how much did each input dimension contribute to the score \(F_c(\mathbf{x})\) for the predicted class \(c\)?
The naive approach is to look at the gradient \(\nabla_\mathbf{x} F_c(\mathbf{x})\): a large partial derivative \(\partial F_c / \partial x_i\) suggests that dimension \(i\) is locally important. The problem is saturation.
2.1.1 Saturation in detail
Many functions used in neural networks (sigmoid, tanh, softmax, squared ReLU for negative inputs) have a characteristic shape: they grow rapidly in the middle range, then flatten out at both ends. The gradient measures the local slope at a single point. When the model has been driven to a very confident prediction, the output sits in one of these flat regions, and the local slope is nearly zero, even if the feature that pushed the model there was crucial.
A concrete example with a sigmoid output \(\sigma(z) = \frac{1}{1+e^{-z}}\), where \(z = \mathbf{w} \cdot \mathbf{x} + b\) is the logit produced by the linear layer. A feature \(x_i\) with a large positive weight \(w_i\) drives \(z\) upward when \(x_i\) is large: the feature is important, the model knows it, and becomes confident. The columns below vary \(z\) to show what happens to the gradient as confidence increases:
| Logit \(z\) | Output \(\sigma(z)\) | Gradient \(\sigma'(z)\) | Gradient × logit |
|---|---|---|---|
| 0.5 (uncertain) | 0.62 | 0.24 | 0.12 |
| 2 (moderately confident) | 0.88 | 0.10 | 0.20 |
| 10 (very confident) | 0.9999 | 0.0001 | 0.001 |
When the logit reaches 10, \(\sigma\) is deep in its flat region: moving from the baseline \(z = 0\) to \(z = 10\) pushed the output from 0.50 to 0.9999 (a contribution of \(\approx 0.50\)), yet the local gradient at \(z = 10\) is \(0.0001\), making the naive attribution 500 times smaller than the actual contribution.
The same issue arises with the softmax at the output of a classifier: once the model assigns probability > 0.99 to a class, the gradient of that probability with respect to any input is tiny, regardless of which tokens actually drove the decision.
Integrated Gradients (Sundararajan et al., 2017) addresses this by accumulating gradients along a straight-line path from a neutral baseline \(\mathbf{x}'\) to the actual input \(\mathbf{x}\), rather than evaluating the gradient at a single point. The path passes through the steep intermediate region where gradients are informative, even if the endpoint is saturated.
2.2 The baseline
The baseline \(\mathbf{x}' \in \mathbb{R}^d\) represents a neutral, uninformative input: a reference point from which the actual input deviates. Choosing the right baseline is problem-dependent:
| Setting | Common baseline |
|---|---|
| Images | All-zero (black) or all-grey image |
| Text (token ids) | Sequence of [PAD] tokens |
| Text (embeddings) | Zero embedding matrix |
In our models, inputs pass through an embedding layer before any computation. Integrated Gradients are computed in embedding space: \(\mathbf{x} \in \mathbb{R}^{T \times d}\) is the embedding matrix of the input text, and \(\mathbf{x}' = \mathbf{0}\) is the zero matrix (equivalent to using a learned PAD-token embedding whose contribution is factored out).
2.3 The integral
The attribution of dimension \((t, j)\) (token \(t\), embedding component \(j\)) is:
\[ \text{IG}_{t,j}(\mathbf{x}) = (x_{t,j} - x'_{t,j}) \times \int_0^1 \frac{\partial F_c\!\left(\mathbf{x}' + \alpha\,(\mathbf{x} - \mathbf{x}')\right)}{\partial x_{t,j}} \, d\alpha \]
Intuitively: we move from \(\mathbf{x}'\) to \(\mathbf{x}\) in small steps \(\alpha \in [0,1]\), evaluate the gradient at each interpolated point \(\mathbf{x}' + \alpha(\mathbf{x} - \mathbf{x}')\), and average those gradients. The factor \((x_{t,j} - x'_{t,j})\) converts gradient units into contribution units (how much did moving from baseline to actual value matter?).
2.4 Completeness: a key axiom
Integrated Gradients satisfy the completeness axiom: the attributions sum exactly to the difference in model output between the input and the baseline.
\[ \sum_{t=1}^{T} \sum_{j=1}^{d} \text{IG}_{t,j}(\mathbf{x}) = F_c(\mathbf{x}) - F_c(\mathbf{x}') \]
This is a conservation law: no contribution is lost or invented. If the model scores the actual input 3.2 points higher than the baseline for class \(c\), the sum of all attributions equals exactly 3.2.
Simple gradient × input does not satisfy completeness in general; Integrated Gradients are the unique path method that does (along the straight-line path).
2.5 Riemann approximation
In practice, the integral is approximated as a Riemann sum with \(m\) steps:
\[ \text{IG}_{t,j}(\mathbf{x}) \approx (x_{t,j} - x'_{t,j}) \times \frac{1}{m} \sum_{k=1}^{m} \frac{\partial F_c\!\left(\mathbf{x}' + \tfrac{k}{m}(\mathbf{x} - \mathbf{x}')\right)}{\partial x_{t,j}} \]
This requires \(m\) forward+backward passes. In practice \(m \in [20, 300]\) is sufficient; larger values reduce approximation error at the cost of compute.
Practical tip: batching interpolated inputs. The \(m\) interpolated inputs can be stacked into a single batch of shape \((m, T)\) and processed in one call, exploiting GPU parallelism. Memory is the binding constraint: for \(m = 300\) and \(T = 128\), the embedding batch is \(300 \times 128 \times d\) floats.
2.6 Token attribution in text
2.6.1 Step 1: embedding dimension → token score
The output of Integrated Gradients is a matrix \(\text{IG} \in \mathbb{R}^{T \times d}\), with one attribution value per token-embedding dimension. To obtain a single token-level importance score, the \(d\) values for token \(t\) must be collapsed to a scalar. Common choices are the \(\ell_1\) norm, the \(\ell_2\) norm, or the signed sum:
\[ \text{score}_{\ell_1}(t) = \sum_{j=1}^{d} |\text{IG}_{t,j}|, \qquad \text{score}_\text{signed}(t) = \sum_{j=1}^{d} \text{IG}_{t,j} \]
The \(\ell_1\) norm treats positive and negative contributions symmetrically; the signed sum preserves direction (a negative score means the token pushed the model away from the predicted class). In our implementation, Captum’s LayerIntegratedGradients already sums over the embedding dimension internally before returning, so we use the signed sum.
2.6.2 Step 2: sub-word tokens → words
WordPiece splits text into sub-word tokens: “cracking” may become [“crack”, “##ing”]. IG produces one score per token, but we want one score per word for readability. The tokenizer returns a word_ids list that maps each token position to its original word index (special tokens such as [PAD] map to None).
The aggregation method we use does the following for each word \(w\):
\[ \text{score}(w) = \sum_{t\,:\,\text{word\_id}(t) = w} \text{score}(t) \]
Sub-token scores are summed (signed sum preserved). This means a word split into \(k\) sub-tokens receives the sum of all \(k\) scores, which slightly inflates the magnitude of rare or long words compared to common ones that tokenize to a single piece.
word_ids list from the tokenizer tracks which token belongs to which word.
2.7 Comparison: Label Attention vs. Integrated Gradients
| Label Attention | Integrated Gradients | |
|---|---|---|
| Type | Intrinsic (built into the model) | Post-hoc (applied after training) |
| Cost | Zero (weights are a free by-product) | \(m\) forward+backward passes |
| Scope | Only for label-attention architectures | Any differentiable model |
| Completeness | No formal guarantee | Yes (by construction) |
| Directionality | Positive only (softmax weights) | Signed (positive & negative) |
| Token granularity | One score per token per class | One score per embedding dim, aggregated to token |