NAF — Single-level classification

1 Hyperparameter sweep

The model predicts only the finest-grained sub-class code (679 classes in the training split) directly, in a single pass, with no auxiliary hierarchy levels — this is the baseline the Multi-level classification page is compared against.

The naf MLflow experiment mixes runs from four different train_fraction sweeps over the same hyperparameter grid: an earlier one at 0.13 (163,615 examples), then 0.26 (314,230 examples), 0.52 (628,461 examples), and the latest, largest one at 1.0 — the full usable set (1,208,579 examples). The corpus itself has 2.3M observations, but only ~1.3M carry a cleaned libelle usable as model input — train_fraction is taken against that ~1.3M subset, not the raw 2.3M. The rest of this section uses 1.0 only (100% of the usable data, the largest and most recent sweep) — the figure below is the exception, comparing all four to isolate the effect of training-data size itself.

Figure 1: Validation loss vs. training set size, by architecture

Growing the training data (163,615 → 314,230 → 628,461 → 1,208,579 examples) steadily improves mean validation loss for both architectures — FastText from ≈0.63 to ≈0.57 to ≈0.51 to ≈0.48, FastText + Label Attention from ≈0.76 to ≈0.63 to ≈0.53 to ≈0.47 — with the label-attention variant benefiting more at each step, consistent with it needing more examples per class to learn meaningful per-class query vectors. The gap between the two narrows from ≈0.13 at 163,615 examples to ≈0.02 at 628,461, and at the full 1,208,579 examples it doesn’t just close — Label Attention actually overtakes FastText (≈0.471 vs. ≈0.482), the only training-size step where the “more moving parts” penalty flips into a genuine advantage.

1.1 Capacity & robustness

Figure 2: Validation loss vs. number of parameters

There’s no clean pareto frontier here — loss doesn’t fall monotonically with parameter count, since num_params mixes together several hyperparameters (vocabulary size, embedding dimension) whose individual effects pull in different directions. The figures further down break each one out separately to make sense of the zigzag.

Label attention wins: mean val loss ≈0.482 for FastText vs. ≈0.471 for FastText + Label Attention. With 679 sub-classes and over a million training examples, each label-attention query gets enough signal to converge into a genuinely discriminative direction, and that pays off in lower loss.

Figure 3: Distribution of validation loss by architecture

The two architectures are equally stable, not just similarly accurate: standard deviation is ≈0.020 for both — no hyperparameter combination pushes either architecture meaningfully off course.

1.2 Effect of vocabulary size and embedding dimension

Figure 4: Effect of vocabulary size on validation loss

FastText traces a clear U — too small a vocabulary is a bottleneck, too large dilutes gradient signal per token, so it bottoms out at a middling vocabulary size before climbing back up. Label Attention doesn’t really share that shape: it only dips slightly at the low end, then climbs steadily as vocabulary keeps growing, without ever curving back down. Label Attention keeps a clear edge over FastText through most of the range, but that gap narrows steadily as vocabulary grows, and the two curves nearly meet once vocabulary gets large — Label Attention’s advantage is at its strongest with a compact vocabulary.

Figure 5: Effect of embedding dimension on validation loss

Both curves dip before rising again — more capacity helps up to a point, then overfitting sets in on the largest embedding table. FastText flattens out quickly and barely moves across the middle of the range, while Label Attention traces a more pronounced U, pulling furthest ahead of FastText at its best embedding size before both converge back toward each other at the largest one.

1.3 Training time

Figure 6: Validation loss vs. training time

Label attention trains markedly slower: mean ≈32,240s (~9.0h) vs. ≈10,460s (~2.9h) for FastText, a ~3.1× gap. With enough data to keep improving, label-attention runs keep going instead of stopping early — and that extra compute buys a genuinely better result, not just parity: label attention has the lower validation loss of the two.

1.4 Interaction: embedding dimension × vocabulary size

Figure 7: Mean validation loss by embedding dimension and vocabulary size

Both knobs interact only mildly. FastText’s best cell (emb_dim=128, vocab=10,000, 0.466) and worst (emb_dim=64, vocab=2,000, 0.497) sit close together, and Label Attention’s grid is similarly compressed: best at (emb_dim=256, vocab=10,000, 0.461), worst at (emb_dim=64, vocab=20,000, 0.485). Label Attention is still somewhat more vocabulary-sensitive at low embedding dimension — growing vocab from 2,000 to 20,000 costs 0.016 at emb_dim=64 versus only 0.012 at emb_dim=128.

2 Prediction analysis

This section examines what the models actually predict, using train_fraction=1.0 and our best run of each architecture. With 679 sub-class targets, a full per-class confusion matrix or example view (as used for Amazon’s 5 classes and CLINC150’s 150) would be unreadable — figures below are adapted to this scale: errors are aggregated to the 21 sections (the coarsest NAF level) wherever a per-class view would be unreadable, and examples show the actual sub-class code and its official label rather than a bare class index.

With 679 sub-classes drawn from a corpus that is nowhere near uniform across them, a natural question is: do errors concentrate in the sub-classes that simply have the fewest training examples? Each point below is one sub-class; x is how many test examples it has (log scale), y is the model’s accuracy restricted to that sub-class.

Figure 8: Per-sub-class accuracy vs. number of test examples for that sub-class (log x-axis)

The funnel shape confirms the hypothesis: below roughly 20 test examples, accuracy swings anywhere from 0% to 100%, but past a few hundred examples it tightens firmly into the 80–100% band. Errors concentrate in the least-represented sub-classes, and the model gets both more accurate and more stable once a sub-class clears a few hundred examples.

The 679-way exact-match accuracy used everywhere else is the strictest possible reading of a wrong prediction — a sub-class miss can still land in the right section, division, group, or class. The two views below unpack this using the NAF hierarchy.

Figure 9: Accuracy by required hierarchy level

The error rate climbs only gradually as the required match gets stricter — low at section level, rising as the granularity demanded gets finer — but most of the story here is how close the two architectures track each other: at every level their error rates sit within a fraction of a point of one another, tracing what is essentially a single shared curve rather than two distinct ones.

Figure 10: Among wrong predictions, the coarsest NAF level at which they still land correctly

Zooming into the wrong predictions only, both architectures split roughly the same way: only ≈7–8% of errors stay within the correct class, and the single largest bucket by far — ≈47–48% — is “No match”, missing even the broad section. The remaining errors are spread fairly evenly across Division (≈20%) and Section (≈15%). So a wrong prediction is about as likely to miss the section entirely as it is to land somewhere within it — the model’s overall accuracy looks stable across granularity, but that average masks two very different failure modes: most predictions are exactly right, and most of the rest are wildly wrong rather than close misses.

The accuracy-vs-support and hierarchical-accuracy views above show that errors happen and roughly how far off they land, but not which sections bleed into which. With 21 sections, a full flow diagram between all of them turns into an unreadable tangle — this view instead ranks the single biggest misclassification flows (correct predictions excluded): which true section, predicted as which other section, accounts for the most errors.

Figure 11: Top misclassification flows between NAF sections (true section → predicted section)

The model only ever predicts the sub-class directly — there’s no separate section/division/group/class head the way the multi-level model has. So “how good is each level” here means truncating that one sub-class prediction into coarser prefixes (exactly like the Hierarchical accuracy tab above) and checking, level by level, whether the truncated guess still lands on the truth.

Figure 12: Example predictions on the NAF test set, across all five hierarchy levels (derived by truncating the sub-class prediction)

3 Explainability

This section turns the spotlight on why the models predict what they predict, not just how often they’re right. The analysis is built around a focused comparison: our best-performing runs of each architecture, FastText against FastText + Label Attention (identical architecture and hyperparameters, mean pooling vs. label attention as the sole difference). Both models are probed with an identical battery of analyses on the same 200 sampled test descriptions, so every figure below is a direct, paired comparison rather than two separate reports glued together.

NAF differs from Amazon (5 classes) and CLINC150 (150 classes) in one important way: 679 sub-classes. Several views used on those pages (a full class × class correlation matrix, one line per class, etc.) would simply be unreadable at this scale, so the figures below either aggregate up to the 21 NAF sections or select a representative subset of sub-classes. There is also no self-attention view here: Amazon and CLINC150 are built on a transformer backbone, but the NAF models compared here are FastText and FastText + Label Attention — bag-of-embeddings architectures with no self-attention layer, so that instrument doesn’t apply.

Three complementary instruments are used throughout:

Layer Integrated Gradients (Captum): a gradient-based, post-hoc attribution method that assigns every input word a signed score for every class: positive = pushes the prediction toward that sub-class, negative = pushes away. It only needs a forward+backward pass through the model, so it can be computed identically for both architectures which is exactly what makes the comparison fair.
Label-attention weights: for the Label Attention model only: the raw cross-attention weights between each class query and the description’s tokens. An explanation that’s built into the architecture, rather than reconstructed after the fact.
Class direction vectors: we compare two 679×emb_dim matrices, one per model: the label-attention queries (label_embeds, 679×emb_dim) from the Label Attention model, and the weight matrix of the linear classification head (linear_weight, 679×emb_dim) from FastText. We examine how the 679 sub-class vectors are arranged relative to each other in each matrix, to see what each architecture’s internal geometry reveals about the classification space.

3.1 Word attributions

Method: Captum (Integrated Gradients) vs. label-attention weights

The fastest way to trust or distrust an explanation is to look at it on a real description. The figure below renders a handful of test descriptions as highlighted text: each word’s background is shaded in proportion to its Captum attribution score. Green = pushes toward the selected sub-class, red = pushes away from it, with colour intensity proportional to |score| / max|score| within that description. When a model predicts incorrectly, you can toggle between predicted and true class to compare what the model thought it was reading versus what it should have read.

Label Attention also produces its own built-in explanation — cross-attention weights, one query per sub-class over the description — shown as a third row (blue intensity = share of attention received). Comparing it against Label Attention’s own Captum row tells you whether this built-in mechanism agrees with the post-hoc explanation of itself: a meaningful check here since, with 679 sub-classes, each query vector gets far less signal per class than Amazon’s 5 classes trained on 200,000 reviews, but considerably more than CLINC150’s 100-example intents — a middle ground consistent with what Training analysis showed above (label attention narrows, but doesn’t fully close, the gap with FastText). This row keeps the same predicted/true toggle described above.

The selection below covers four informative cases: descriptions where the two architectures disagree on the predicted sub-class (the richest source of qualitative insight: what did one model see that the other didn’t?); a couple where they agree and are correct (a sanity check on what a “good” explanation looks like); cases where both are wrong (what misleads them?); and cases where FastText is correct but FastText + Label Attention is not (when does the simpler architecture do better?).

1 / 9

true sub-class: 6630Y — Activités de gestion de fonds — “toutes opérations mobilières financières et de prestations de toute nature avec toutes sociétés filiales apparentées ou dépendant du mêm”

Label Attention — Captum IG → predicts 8210Y — Activités de service de bureau et de soutien administratif (✗ incorrect)

predicted class