CLINC150
CLINC150 is an intent detection dataset with 150 intent classes covering everyday conversational domains (banking, travel, home automation, etc.), with 100 training examples per class. The large number of classes and the short, colloquial nature of utterances make it a natural intermediate step between Amazon and NAF.
Two training regimes were explored: the standard 150-class setup and an extended version that adds an out-of-scope (OOS) class grouping utterances that do not match any of the 150 intents. Both regimes produce similar training behaviour — loss curves and hyperparameter sensitivities are nearly identical, with overall performance slightly lower in the OOS setting. The interesting differences emerge at prediction time and will be covered in a dedicated subsection.
1 Hyperparameter sweep
This section explores the results across four complementary angles: model capacity and robustness; the effect of vocabulary size and embedding dimension; the effect of the number of attention heads; and training time.
1.1 Capacity & robustness
A key observation is that n_head and lr do not affect num_params. In a standard transformer, the total size of the attention matrices is \(4 \times d^2\) regardless of how many heads are used, since \(d_{\text{head}} = d / n_{\text{head}}\). Similarly, lr is a training hyperparameter with no effect on architecture. As a result, for any fixed combination of (emb_dim, n_layers, vocab_size), FATE architectures have exactly \(3 \times 3 = 9\) runs sharing the same num_params (3 values of n_head × 3 values of lr), while FastText has only \(3\) runs. This motivates two complementary views:
- Scatter plot: all 504 runs, each point being that run’s
min_val_loss(the minimum over its training epochs), showing the full spread of achievable val_loss for a given parameter budget. - Curve plot: for each
(architecture, emb_dim, vocab_size)group, take themin(val_loss)overn_headandlr, tracing the performance frontier as model capacity grows.
The most striking observation is the clear split into two performance tiers, which is much sharper than on Amazon. The lower tier (val loss ≈ 0.3–0.5) contains FastText, FATE (L=1), and FATE (L=2); the upper tier (val loss ≈ 0.7–1.4) contains their Label Attention counterparts. On Amazon, label attention had a small but consistent positive effect — here it consistently hurts. A likely explanation: Label Attention must learn one query vector per class — a representation of “what this class looks for in the text”. With only 100 examples per class, each query receives too little gradient signal to converge to something meaningful and ends up fitting noise. Mean Pooling is unaffected: it requires no per-class learning at the aggregation step, so all gradient flows directly to the embedding and the linear head. On Amazon (200k examples, 5 classes), the queries have sufficient signal and the class-specific attention mechanism provides a consistent gain.
The violin plot shows the distribution of validation loss across all runs for each architecture. A narrow violin indicates robustness to hyperparameter choice; a wide one signals high sensitivity.
The two-tier structure from the previous plot reappears here. The three architectures without label attention (FastText, FATE L=1, FATE L=2) all produce tight, low-variance violins centered around 0.38, confirming they are robust to the choice of lr and n_head. Their label attention counterparts are not only worse on average but dramatically more sensitive: the violins span 0.7 to 1.7+, meaning a poor hyperparameter choice can degrade performance by nearly a factor of four. FATE (L=2) + Label Attention is the least erratic of the three, but still far above the non-LA tier.
1.2 Effect of vocabulary size and embedding dimension
Note: the three nominal vocabulary sizes tested are 2,000, 5,000, and ≈8,407. The last value was obtained by targeting 10,000 tokens, but CLINC150’s short conversational utterances do not contain enough distinct surface forms to fill a vocabulary that large. Each run trains its own WordPiece tokenizer independently, and the exact saturation point varies slightly (8,406–8,408) due to non-determinism in the tokenizer training algorithm (tie-breaking of merge pairs, iteration order). Since this variation is an artefact of the tokenizer rather than a meaningful hyperparameter difference, all runs with a 10,000 target are collapsed to their average (8,407) and treated as a single vocabulary size throughout this analysis.
The trends are somewhat counterintuitive: for non-LA architectures, performance slightly degrades as vocabulary grows. With only 15,000 training examples, a large vocabulary creates a sparse coverage problem. The embedding layer is a lookup table: gradients are computed only for tokens present in the current batch — absent tokens receive zero gradient. At ~8,400 vocab, many tokens appear only a handful of times in the entire corpus, so their embedding vectors are updated too rarely to converge to anything meaningful. A compact 2,000-token vocabulary avoids this: rare words are decomposed into frequent sub-tokens that accumulate enough gradient signal across batches, and the embedding table as a whole is used far more densely. On Amazon, the 200k examples provide enough coverage to support larger vocabularies; CLINC150’s 15k examples cannot.
Embedding dimension has a clear and consistent effect: all architectures improve as emb_dim grows, with the steepest gains between 64 and 256. Beyond 256, returns diminish but do not vanish — 512 still frequently outperforms 256, meaning the model continues to benefit from a richer representation even at higher capacity. 150 fine-grained intents genuinely require more expressive embeddings than Amazon’s 5 sentiment classes. The two performance tiers remain perfectly separated at every emb_dim value: no amount of embedding capacity compensates for the overfitting introduced by label attention on this small dataset.
1.3 Effect of number of attention heads
The number of attention heads has virtually no effect on validation loss — the mean lines are flat across 2, 4, and 8 heads for all four architectures. This is consistent with what was observed on Amazon and confirms the theoretical expectation: since total attention capacity is fixed at \(4d^2\) regardless of head count, the head split is a structural choice rather than a capacity one. On CLINC150 specifically, the task may not require the kind of multi-perspective token interactions that more heads are designed to capture — a single or few-headed attention over short utterances is sufficient. Whether heads develop specialised attention patterns despite similar loss values may become apparent in the self-attention matrix analysis.
1.4 Training time
The plot is dominated by the same two-tier structure seen throughout this analysis. The lower cloud (val loss ≈ 0.35–0.5) groups all three non-LA architectures — FastText, FATE (L=1), and FATE (L=2) — across a wide range of training times (roughly 100s to 7k s). Within this cloud the three architectures are largely interleaved: there is no clean ordering where more compute reliably buys better performance, consistent with the observation that the dominant driver of quality is architecture choice rather than training budget. The upper cloud (val loss ≈ 0.7–1.8) contains exclusively the label attention variants, and they are noticeably shifted to the left — they train faster than their non-LA counterparts. This is not an efficiency gain: it is a consequence of early stopping. Because label attention models fail to improve their validation loss, the patience threshold is reached sooner and training terminates early. Less compute is spent, but as a symptom of poor convergence rather than architectural efficiency.
1.5 Interaction: embedding dimension × vocabulary size
The heatmaps consolidate the two previous findings in a single view. emb_dim dominates: the vertical gradient is strong and consistent across all six panels — going from 64 to 512 cuts val loss by ~0.07 for non-LA architectures and by ~0.15–0.3 for LA ones. Vocabulary size has a weaker but consistently negative effect: the horizontal gradient is mild, but every panel shows a slight left-to-right darkening, confirming that 2,000 tokens is the best choice regardless of architecture. The two effects appear largely additive — there is no notable interaction between them. A striking result at the optimum (emb_dim=512, vocab=2,000) is that FastText (0.342), FATE (L=1) (0.343), and FATE (L=2) (0.344) are essentially indistinguishable: once the embedding table is large enough and the vocabulary compact, the transformer layers bring no additional gain on this dataset.
2 Prediction analysis
This section examines what the models actually predict — where they agree, where they fail, and which classes are structurally hard. For each of the six architectures, predictions are taken from the best-performing run (lowest validation loss). As noted above, the OOS regime produces similar training dynamics; the interesting differences lie at inference time and are covered in the dedicated subsection.
2.1 No-OOS results
Unlike Amazon’s 1★–5★ ratings, CLINC150’s 150 intents have no inherent order, so “prediction distance” can’t mean a numeric gap between labels the way it does there. The closest meaningful analogue is domain distance: is a wrong prediction at least in the right domain (e.g. mixing up two banking intents), or does it miss the domain entirely?
Label attention roughly doubles the “wrong domain” share versus its non-attention counterpart at every capacity level (FastText: 5.0%→9.8%; FATE (L=1): 4.1%→8.5%; FATE (L=2): 3.7%→7.1%) — consistent with its lower overall accuracy, but the gap is entirely in the worst error bucket, not the “same domain” one, which barely moves (3.6–3.8% without label attention vs. 5.6–6.6% with it). When label attention gets it wrong on CLINC150, it’s more likely to miss the domain entirely rather than land on a plausible near-miss.
The most frequent confusions are mostly structurally related intents: pairs sharing vocabulary or near-duplicate intent definitions like calendar / calendar_update, bill_balance / bill_due, or order / order_status. This kind of confusion is largely intrinsic to the task and unlikely to be resolved by scaling the model.
Rather than asking which model is wrong, this view asks: when models disagree, is that because some are weaker, or because the utterance itself is ambiguous? For each test example we count how many distinct intent predictions the three models within each group produce, then look at how often the majority vote is correct at each agreement level. The analysis is run separately for the non-LA and LA groups.
Note: when all three models predict different classes (“3 distinct”), there is no true majority. The vote is broken by taking the lowest class index among the three predictions — the accuracy shown in that column should therefore be interpreted with caution.
Each row is a domain; each cell is one intent class, coloured by its mean per-class accuracy averaged across the six architectures. Green cells are easy classes (all models agree and are correct); red cells are universally hard ones. White/grey gaps simply pad shorter domains to the same width.
2.2 Comparison: with OOS vs. without OOS
This section compares the two training regimes head-to-head: models trained on the full dataset including the out-of-scope class (151 classes) versus models trained without it (150 classes). All comparisons are restricted to in-scope examples only so the evaluation target is identical in both cases.
Same format as the confusion matrix in the Predictions section, with an extra “OOS” row (true OOS examples) and “OOS” column (examples predicted as OOS). The diagonal entry of the OOS row shows how well the model identifies actual out-of-scope utterances; the OOS column reveals which in-scope domains are most contaminated.
Label attention brings no in-domain upside to offset its OOS cost: the mean in-scope diagonal is if anything slightly lower with label attention than without — it loses on both axes, not just OOS recall. The real differentiator is OOS recall (the OOS→OOS cell): FATE without label attention detects out-of-scope utterances best (56% at L=2, 52% at L=1), FATE+LA falls markedly (38% at L=2, 20% at L=1), and FastText struggles regardless of label attention (25% plain, 19% with LA) — a plain mean-pooling model with no self-attention layers at all appears to lack the contextual signal needed to recognise atypical utterances, so weak OOS detection isn’t purely a label-attention problem.
Training with OOS barely moves in-scope accuracy for any architecture — the with/without pairs are nearly identical, deltas stay within ±0.02 in either direction (FATE (L=1)+LA is the only one that improves, +0.02). The dominant pattern in this chart isn’t the OOS regime at all: it’s label attention. Every label-attention variant sits 5–9 points below its non-LA counterpart, regardless of whether OOS was included in training — confirming that label attention’s cost is structural, not something introduced or worsened by the OOS class.
Each row shows one of the 10 most OOS-contaminated classes. The orange dot is the per-class accuracy of the model trained with OOS; the blue dot is the accuracy of the model trained without OOS. The red annotation gives the contamination rate — the share of that class’s test examples that the with-OOS model incorrectly predicted as OOS instead of the correct intent.
Two distinct patterns emerge. For classes where the two dots overlap (e.g. book_flight, spending_history, measurement_conversion), contamination exists but does not translate into an accuracy drop: even though a fraction of examples get absorbed by the OOS catch-all, the model still classifies the remainder correctly, and the overall per-class accuracy is unaffected. For classes where the orange dot sits clearly to the left of the blue (e.g. distance, change_ai_name), the with-OOS model has systematically learned to treat these intents as out-of-scope, causing a substantial accuracy loss that disappears once the OOS class is removed from training. These are the intents whose surface forms — generic, open-ended queries — overlap most with what a real OOS utterance looks like, making them vulnerable to the OOS attractor.
This chart isolates examples where adding OOS training directly causes a mistake — i.e. the no-OOS model is correct but the with-OOS model is wrong on the same example. Among those cases, it answers: why does the with-OOS model fail?
- Absorbed by OOS class (red): the model explicitly routes the example to the OOS catch-all. The OOS class acts as a drain that pulls in-scope utterances. This is a direct effect of OOS training — the model has learned to treat certain in-scope phrasing as out-of-scope.
- Confused with another in-scope class (orange): the OOS class is not the predicted label, but training with it has still shifted the decision boundaries enough to break a classification that was previously correct. This is an indirect effect — the in-scope label space has been reorganised by the presence of the OOS attractor during training.
The balance between the two bars tells us whether OOS contamination is primarily a direct routing problem or a more diffuse boundary distortion.
3 Explainability
For each architecture, the best-performing model — the run with the lowest min_val_loss — is used throughout this section. Four complementary instruments are used:
- Layer Integrated Gradients (Captum) : a gradient-based, post-hoc attribution method that assigns every input word a signed score for every class: positive = pushes the prediction toward that intent, negative = pushes away. It only needs a forward+backward pass through the model, so it can be computed identically for both architectures which is exactly what makes the comparison fair.
- Label-attention weights : for the Label Attention model only: the raw cross-attention weights between each class query and the utterance’s tokens — an explanation built into the architecture, rather than reconstructed after the fact.
- Self-attention matrices : both models are built on the same transformer backbone, so we can also peek at how the encoder itself distributes attention across the utterance, layer by layer, before any classification happens.
- Class direction vectors : we compare two 150×256 matrices, one per model: the label-attention queries from the Label Attention model, and the weight matrix of the linear classification head from the Mean Pooling model. We examine how the 150 intent class vectors are arranged relative to each other in each matrix, to see what each architecture’s internal geometry reveals about the intent space.
3.1 Word attributions
Method: Captum (Integrated Gradients) vs. label-attention weights
The fastest way to trust or distrust an explanation is to look at it on a real utterance. Each card below renders one test utterance as highlighted text: Mean Pooling’s and Label Attention’s Captum scores, shaded green = pushes toward the selected intent, red = pushes away from it. Comparing these two rows tells you whether the two architectures rely on the same words.
Label Attention also produces its own built-in explanation — cross-attention weights, one query per class over the utterance — shown as a third row (blue intensity = share of attention received). Comparing it against Label Attention’s own Captum row tells you whether this built-in mechanism agrees with the post-hoc explanation of itself — a meaningful check given that label attention hurts on CLINC150, where 100 training examples per class may not be enough for its 150 query vectors to converge to discriminative directions. All three rows share one class selector (top-10 by the model’s own confidence ranking, true intent always included), so switching class updates every row at once.
The selection covers four informative cases: utterances where the two architectures disagree on the predicted intent; a couple where they agree and are correct; cases where both are wrong; and cases where Mean Pooling is correct but Label Attention is not.
3.2 Corpus-level word importance
Method: Captum (Integrated Gradients)
Zooming out from individual utterances: which words does each architecture rely on in general for each domain? Rather than displaying 150 per-class charts, we aggregate the Captum scores across all classes within each of the 10 CLINC150 domains, keeping only words seen at least 3 times. This gives a compact view of the vocabulary each domain relies on — and whether the two architectures agree on what matters.
3.3 Distinctiveness of per-class explanations
Method: Captum (Integrated Gradients)
For each architecture, every intent’s per-word mean IG scores (from the previous section) form a length-|vocabulary| profile vector; the Pearson \(r\) between every pair of these 150 vectors gives a full 150×150 correlation matrix. That matrix is unreadable as a heatmap, so we select 20 intents according to a precise criterion and display only that 20×20 slice. Two such selections are shown below, each answering a different question. Use the buttons to switch between the two architectures.
3.3.1 Most-confused intents
We select the 20 intents with the highest error rate, as selected by the Label Attention model, and display the 20×20 sub-matrix of their word-attribution profiles. If the model confuses intents A and B, their attribution profiles should look similar (high positive correlation). Hot off-diagonal cells reveal intent pairs that are simultaneously hard to classify and hard to distinguish in attribution space.
3.3.2 Most similar intent pairs
We identify the 20 intents involved in the highest-correlation pairs (highest off-diagonal r in the Label Attention matrix). These are intents for which the model produces nearly identical explanations despite different semantics — potential blind spots. The sub-matrix reveals their similarity structure.
3.4 Self-attention and label attention: how each mechanism reads the utterance
3.4.1 Self-attention
Method: raw self-attention weights
Both architectures are built on the same transformer backbone : they only differ in how its token-level outputs get turned into a classification decision. For the same short utterance, we extract every layer’s per-head self-attention matrices, for both models, side by side.
Reading an attention heatmap: each cell (i, j) shows how much token i attends to token j when building its contextual representation (rows sum to 1). CLINC150 utterances are short (typically 5–15 words), so the matrices are small and easy to read. A strong diagonal indicates local, syntax-like processing; vertical stripes (attention sinks) indicate tokens that many others converge onto; a diffuse matrix means global mixing across the whole utterance.
Per-head self-attention — "i need some directions to phoenix"
true: directions · Label Attention → tell_joke ✗ · Mean Pooling → directions ✓
true: directions · Label Attention → tell_joke ✗ · Mean Pooling → directions ✓
In Label Attention, Layer 1 stays diffuse across all four heads, with no consistent token to settle on. By Layer 2, a couple of heads start to sharpen and lean toward content words, but the rest stay diffuse — the architecture never fully commits to a hub token.
Mean Pooling shows the same progression, only earlier. Already in Layer 1, the majority of heads behave like dedicated summarizers, fixating on the utterance’s key content word almost regardless of what the sentence is about, even though a few stay diffuse. By Layer 2, everything converges: all four heads now share this same content-focused behaviour.
A plausible explanation is that Mean Pooling has to pre-aggregate the utterance’s information onto a handful of tokens before the unweighted average is taken, whereas Label Attention can leave that selection to its own dedicated cross-attention step downstream — which is why its self-attention rarely needs to commit to a single hub token. Unlike Amazon, though, we don’t see this pay off as a clean local pattern: only a couple of heads lean toward content words by Layer 2, most stay diffuse — likely because with only two layers (vs. Amazon’s four) and ~100 examples per intent, the label-attention queries may not have converged to a discriminative enough target to offload onto.
3.4.2 Label attention
Method: raw label-attention weights
The label-attention module has 4 heads, each producing its own (class × number of words in the utterance) attention matrix. The mean over heads shown below is purely a visualisation aid for this explanation — it is not a computation the model itself performs. Given that label attention hurts on CLINC150, it is worth checking whether the heads have learned anything meaningful or whether their attention is diffuse and unstructured.
With 150 intents, each figure below only shows the 10 most relevant: the predicted and true intent, plus the classes the model’s final softmax score ranked highest for this utterance — i.e. the classes the model actually considered, rather than classes picked from the attention weights themselves. Unlike an attention-based ranking, this selection doesn’t depend on the head, so the same 10 classes appear in all 5 panels (4 heads + mean), making them directly comparable.
The 4 heads generally attend to different words from one another rather than duplicating each other, but none of them settles into a sharp, decisive pattern — all stay comparably diffuse, spreading weight across several words instead of committing to one. The heads split up what they look at without any single one acting as a clear specialist.
3.5 Are these explanations faithful? A word-deletion stress test
Method: Captum (Integrated Gradients)
Attribution scores are only meaningful if the model would actually change its mind were those words removed. We test this directly: progressively delete the words Captum ranked as most important for the predicted class, and watch how fast that class’s predicted probability collapses. As a control, we delete the same number of randomly chosen words, averaged over 3 draws, isolating the effect of which words are removed from the trivial effect of mutilating the utterance.
On short utterances (5–15 words), this test is particularly demanding: even removing 1–2 words is a large fraction of the input, so both curves can drop steeply. The key is whether the gap between guided and random removal is visible — a non-trivial gap confirms that the attribution method identifies words the model genuinely depends on, not just any words.
All three guided curves collapse far faster than their random controls — the gap peaks around 25% of words removed (+0.55 for Label Attention’s Captum scores, +0.59 for its own attention weights, +0.55 for Mean Pooling), confirming both attribution methods identify words the models genuinely depend on rather than merely correlate with. Label Attention’s built-in attention weights are, if anything, the most faithful guide of the three — its curve drops fastest and lowest (0.045 retained at 50% removed, vs. 0.104 for its own Captum scores and 0.075 for Mean Pooling), a reassuring sign that its own explanation isn’t decorative.