This page compares against the Single-level classification baseline, described on its own page.
The five-level hierarchy of NAF 2025 opens a natural multi-task training strategy: rather than predicting only the finest level (sub-class), the model can be trained to simultaneously predict multiple levels of the hierarchy. Two theoretical benefits follow: (1) coarser levels are easier to classify and inject additional gradient signal that guides the shared representation towards features discriminant across the taxonomy; (2) at inference time the model directly outputs predictions at every level, requiring no post-hoc aggregation.
All five NAF levels are used: the section (1 letter, 20 codes in the training set out of 21 in the full nomenclature), the division (first 2 characters, 84 out of 87), the group (first 3 characters, 274 out of 287), the class (first 4 characters, 586 out of 651), and the sub-class (full 5-character code, 679 out of 747). All five classification heads share a single TokenEmbedder — one WordPiece tokenizer and one embedding matrix — that is jointly optimised through the sum of five independent cross-entropy losses. Each level then has its own SentenceEmbedder (either mean pooling or label attention) and a ClassificationHead (a single linear projection). This design keeps the parameter count modest: only the five separate sentence embedders and linear heads add level-specific parameters on top of the shared trunk. The architecture natively supports additional categorical input variables; they are not used here for simplicity, as the NAF dataset provides only a free-text description.
A word of caution on “independent”: the five heads never talk to each other. Nothing forces the section, division, group, class, and sub-class predictions for the same example to be mutually consistent — they are five separate softmaxes over five separate label spaces, each optimised against its own ground truth, with no shared constraint tying them to the same underlying NAF code. On Single-level classification, by contrast, only the sub-class is ever predicted, and every coarser level is a deterministic truncation of that one code — so its levels are always mutually consistent by construction, and can never disagree with each other. Here, they can, and do: it is entirely possible for the division head to land on the correct division while the section head (independently) misses, even though the true division determines the true section — an outcome that is mathematically impossible under truncation. This isn’t a flaw to fix; it’s the entire point of giving each level its own head — but the five predictions shown anywhere on this page should be read as independent opinions, not a single coherent decision. Sharing one token encoder trained on the sum of all five losses could still nudge the heads toward agreeing in practice; whether it actually does is an empirical question the results further down answer directly.
Multi-level NACE Classification — Architecture
Input
"Conseil en stratégie et management"
(B,) · str
Shared Token Encoder — weights shared across all levels
(B, L, d) · float32
────►
(B, L, d) · float32
────►
T
Token embeddings
reused by every head
Level 1 · Section
∑÷n
SentenceEmbedder
Mean Pool or Label Attn
(B, d) or (B, C, d)
ℒsec
· · ·
Level k · …
∑÷n
SentenceEmbedder
Mean Pool or Label Attn
(B, d) or (B, C, d)
ℒk
· · ·
Level 5 · Sub-class
∑÷n
SentenceEmbedder
Mean Pool or Label Attn
(B, d) or (B, C, d)
ℒsub
Joint training loss
ℒ = ℒsec + ℒdiv + ℒgrp + ℒcls + ℒsub
Independent cross-entropy at each level — summed and back-propagated through the shared trunk
Class counts (20 / 84 / 274 / 586 / 679) reflect codes observed in the training split (train_fraction = 0.13 of the full corpus). Full NAF 2025: 21 / 87 / 287 / 651 / 747.
Hyperparameter sweep
Architecture is fixed here (n_layers=0, i.e. FastText-style mean pooling — the multi-level trainer does not yet expose the FATE transformer variant), so the sweep covers embedding_dim in {64, 128, 256, 512} × vocab_size in {2000, 5000, 10000, 20000} × n_heads_label_attention in {none, 4} × lr in {0.0005, 0.001} — lr=0.0001 was dropped from this grid, giving 4 × 4 × 2 × 2 = 64 runs per train_fraction rather than 96. As with the Single-level classification page, the naf_multilevel MLflow experiment mixes four train_fraction sweeps over the same grid: 0.13, 0.26, 0.52, and 1.0. The rest of this section uses 1.0 only (the full, largest sweep) — the figure below is the exception, comparing all four to isolate the effect of training-data size itself.
Growing the training data (163,615 → 327,230 → 654,461 → 1,208,579 examples) steadily improves mean validation loss for both architectures — FastText from ≈0.537 to ≈0.491 to ≈0.457 to ≈0.439, FastText + Label Attention from ≈0.723 to ≈0.591 to ≈0.492 to ≈0.440 — with the label-attention variant benefiting more at each step, the same pattern as Single-level classification: the gap between the two architectures narrows from ≈0.186 at 163,615 examples to ≈0.100 at 327,230 to ≈0.035 at 654,461 to essentially zero (≈0.002) at 1,208,579, as label attention gets enough examples per class to learn meaningful per-class query vectors — here shared jointly across five hierarchy levels rather than one — and fully closes the gap with plain FastText once the full training set is used.
Note: the two middle sweeps land on slightly different example counts than Single-level classification’s own 314,230 / 628,461 (both endpoints, 163,615 and 1,208,579, match exactly) — most likely a small difference in which corpus snapshot each intermediate sweep ran against, not a discrepancy in how multi-level filters examples.
Capacity & robustness
As on the single-level page, there’s no clean pareto frontier — num_params mixes together vocab_size and emb_dim, whose individual effects pull in different directions (see the interaction heatmap further down). Label attention is on par with FastText on average here (≈0.441 vs. ≈0.439, a ≈0.002 gap — statistically indistinguishable) and its best configuration even beats FastText’s best (≈0.417 vs. ≈0.426). But it’s noisier: at its largest configuration (emb_dim=512, vocab_size=20000, ≈16.3M params) it spikes to the worst validation loss of either architecture (≈0.467), reproduced across both runs there — extra capacity seems to overfit rather than help.
FastText is also more stable: its results cluster tightly (std ≈0.0081) while Label Attention spreads almost twice as wide (std ≈0.0150) — the same direction as the single-level page. Medians are nearly identical (≈0.439 vs. ≈0.440), so this spread, not the average, is what separates the two architectures here.
Effect of vocabulary size and embedding dimension
FastText is fairly flat across vocabulary size (0.4464→0.4363→0.4343→0.4376 from 2,000 to 20,000 tokens, best at 10,000), while FastText + Label Attention dips to its best at 5,000 (0.4328) before degrading sharply toward 20,000 (0.4541) — a sparse-coverage effect that only bites past a certain vocabulary size, compounded across five classification heads sharing one token embedder.
Both architectures follow a mild U-shape, best around emb_dim=256 (FastText: 0.4444→0.4350→0.4345→0.4408 from 64 to 512; Label Attention: 0.4524→0.4393→0.4346→0.4356) — the smallest embedding (64) is consistently the worst choice for either architecture, and label attention keeps improving up to 512, coming within 0.001 of its best score there.
Training time
Label attention trains dramatically slower here — mean ≈53,930s (≈15.0h) vs. ≈9,520s (≈2.6h) for FastText, roughly a 5.7× gap: five heads sharing one label-attention-driven sentence embedder mean five times the cross-attention computation per forward pass, and with a full training set to keep improving on, the run rarely stops early, paying that per-step cost for the whole training budget.
Interaction: embedding dimension × vocabulary size
FastText is fairly flat across the grid — best cell (emb_dim=128, vocab=10,000, 0.430) and worst (emb_dim=64, vocab=2,000, 0.455) are close together. Label Attention is more sensitive, but not uniformly: at emb_dim=64, growing the vocab from 2,000 to 20,000 barely moves the loss (0.450→0.458, a 0.008 swing), while the same vocab change at emb_dim=512 swings much more (0.440→0.463, a 0.022 swing) — the single worst cell in the whole grid. A larger token embedder doesn’t buy label attention robustness to vocabulary size here; if anything it makes the model more sensitive to it.
Prediction analysis and comparison with single-level
This section examines what the models actually predict, using the same two architectures as Single-level classification — FastText and FastText + Label Attention. For each architecture, the model is the single run from the hyperparameter sweep above with the best sub-class test accuracy — not the lowest overall (weighted) validation loss used to pick “best” in the sweep analysis — since sub-class is the finest, most demanding level and the one directly comparable to the single-level page. Unlike that page, exact-match accuracy on the 679 sub-classes is not the only thing available here: the model predicts all five hierarchy levels directly, through five independent classification heads sharing one token encoder, rather than deriving coarser levels by truncating a single sub-class prediction. On the single-level page, only the sub-class code is predicted (e.g. 6820G); the coarser levels are never predicted themselves, just mechanically cut from that one string (code[:2] for the division, code[:4] for the class, etc.). Here, each level gets its own dedicated prediction, so — unlike a truncated code, which is right at every coarser level whenever the sub-class is right — the five levels can disagree with each other (e.g. a correct section with a wrong sub-class).
Same question as on the single-level page, restricted to the finest (sub-class) head: do errors concentrate in the sub-classes with the fewest training examples?
The same funnel shape as on the single-level page: 311 of the 513 sub-classes present in the test set have fewer than 20 test examples, and their accuracy swings across the entire 0–100% range; the 50 sub-classes with 200+ examples tighten into a 56–100% band (mean ≈89%). Sharing the token encoder with four auxiliary heads doesn’t change this — sub-class accuracy is still driven by how much data that specific sub-class has, not by the auxiliary supervision.
An earlier version of this section derived every coarser level by truncating the sub-class head’s prediction, exactly like Single-level classification is forced to (it has no separate section/division/group/class head to ask). That method turned out to be misleading here: the multi-level model has four genuinely independent auxiliary heads, each trained with its own cross-entropy loss, and they are not reducible to a truncation of the sub-class guess. Checking directly — among examples where the sub-class head is wrong, how often is the independent section head still correct — gives 61.4–61.5%, against only 52.4–53.0% from truncating the (wrong) sub-class guess. The charts below query each head directly.
The gap isn’t monotonic in size, but it never reverses: truncating the single-level model’s sub-class guess leads multi-level’s own independent heads at every level, from over a point ahead at Section down to an exact tie at Sub-class, narrowing steadily through Division, Group and Class rather than crossing over. That convergence is worth flagging: single-level and multi-level are two separately trained models, not the same model with heads removed, so there’s no reason to expect them to land this close even on the one task (sub-class) both nominally solve directly — whatever benefit the auxiliary per-level losses provide to multi-level’s coarser heads, they don’t appear to give its own sub-class head any edge over a model trained on that task alone.
The next view zooms into the wrong sub-class predictions only, and buckets each one by the finest of Class/Group/Division/Section that still lands correctly. For single-level, that bucket is unambiguous — truncation is nested by construction, so a correct division always implies a correct section too (section is deterministically derived from division), and conversely a wrong section forces division, group and class to be wrong as well. For multi-level, it’s less obvious than it sounds: “No match” means all four independent heads miss at once, not just the section head — since the four heads are separately trained and never consult each other, it’s entirely possible for the division head to land correctly while the (equally independent) section head misses the very same example, an inconsistency truncation can never produce.
The single-level bars (hatched) confirm this isn’t just an artefact of the earlier derived comparison: on Single-level classification, ≈47–48% of errors are “No match” (not even the section is right) for both architectures, and only ≈7–8% stay within the correct class. Multi-level’s independent heads do much better on both counts: “No match” is only 34.0% for FastText and 28.9% for Label Attention, and — unlike the single-level bars, which look nearly identical to each other — the two architectures now diverge meaningfully, with Label Attention landing in the Class bucket far more often (25.1% vs. 11.3% for FastText). When its finest guess is wrong, its independently-trained class head is much more likely to still be right, even though its raw sub-class accuracy is slightly lower. The auxiliary heads are doing real work that the single-level bars, sitting right next to them, simply cannot do.
The buckets in the previous tab already prove the auxiliary heads are doing real work, but they still read like an abstract accounting exercise. This tab isolates the mechanism directly — the single-level model is not involved here, both sides of the comparison come from the multi-level model alone: restricted to the exact same wrong-sub-class examples, how often does truncating that same model’s own wrong sub-class guess land on the truth, versus how often does its independent head get it right on its own?
For both architectures and both levels, the grey (truncated) bar is consistently well below the coloured (direct) bar — the independent head recovers real accuracy that a truncation-only view throws away: +8.6pp at section / +9.6pp at division for FastText, +9.9pp / +13.1pp for Label Attention. This is the concrete mechanism behind the previous tab’s better “No match” rate: it isn’t that the model becomes more accurate overall, it’s that its mistakes are genuinely localised to one head — the sub-class head can misfire on the fine-grained decision while the division and section heads, trained with their own independent loss on the same shared token encoding, are unaffected and still answer correctly.
Why not group and class too? 576 of the 651 NAF classes (88.5%) contain exactly one sub-class — for those, getting the class right and getting the sub-class right are the same event, so whenever the sub-class head is wrong, truncating that wrong guess to class level is necessarily also wrong: truncated accuracy collapses to ~0% at those two levels, which would make the comparison trivial rather than informative. Section and division are coarse enough that many different sub-classes map to the same coarser code, so truncation and direct prediction can genuinely diverge — that’s what the chart above needs to be meaningful.
The two largest flows both involve section M (scientific & technical activities) — confused with K (financial & insurance, 7.0% of FastText’s section-level errors) and J (information & communication, 6.2%), with the reverse flow J→M also in the top three (6.2%). This is a symmetric, semantically coherent confusion: consulting, IT, and technical-services activities genuinely overlap in vocabulary with finance and information-services activities, and it shows up as a real bidirectional flow rather than a one-way drain into a single dominant class — unlike the OOS “catch-all” pattern seen on CLINC150.
Browsing through these cards turns up the independent-heads quirk from the top of the page in concrete form: it’s not rare to see the sub-class head land exactly right while the (independently predicted) section head misses — a combination truncation could never produce, since a correct sub-class would truncate down to a correct section every time. Here, the two heads simply disagree, each having made its own call from the shared encoding.
Explainability
This section asks why the multi-level model predicts what it predicts — and, unlike Single-level classification, it can ask that question five times over, once per independent head. The same description now gets five separate explanations (section, division, group, class, sub-class), which opens a question single-level explainability can’t: does the model rely on different words to justify a coarse call than a fine one? And when a “smart mistake” happens — right section, wrong sub-class, as seen throughout the Prediction analysis section above — what did each head actually look at?
The comparison is again between FastText and FastText + Label Attention, each level’s own independently-trained head, on the same 200 sampled test descriptions.
Three complementary instruments, same as on the single-level page:
- Layer Integrated Gradients (Captum): word-level attribution, computed independently for each of the 5 heads. Run without any class cap here — the cap used for label attention below wasn’t needed for Captum, whose arrays carry no per-head dimension.
- Label-attention weights (Label Attention model only): the model’s own cross-attention, also computed per level. Unlike Captum, this one does carry an extra head dimension (4 attention heads per level’s mechanism) on top of the class dimension — summed across 5 levels’ worth of ~1700 classes, the raw version was too large to log reliably, so it’s capped to each level’s top-100 predicted classes (no cap actually bites at section or division, which have fewer than 100 classes to begin with).
- Class direction vectors: per level, the label-attention queries or the linear head’s weight rows — five geometries to compare instead of one.
Word attributions across the hierarchy
Method: Captum (Integrated Gradients)
For a single description, each level’s head reaches its own verdict from the same shared encoding — the figure below renders all five as highlighted text, stacked, using that level’s own attribution for whatever class it actually predicted. Green pushes toward that level’s prediction, red pushes away, colour intensity ∝ |score| / max|score| within that row.
The examples below are deliberately picked to include the “smart mistakes” quantified earlier in Prediction analysis — right section, wrong sub-class — alongside a couple of clean sweeps (every level correct) and clean misses (every level wrong), so the shift in emphasis from coarse to fine rows is easy to compare.
Corpus-level word importance, direct at the section level
Method: Captum (Integrated Gradients)
On Single-level classification, the equivalent figure had to approximate section-level importance by grouping together the attributions of whichever sub-classes happen to fall under each section — there was no section-level prediction to ask directly. Here there is: the section head is independently trained, so the word importance below comes straight from it, for the 6 best-represented sections in the test sample. As on that page, use the buttons above the figure to switch between the two architectures.
Label attention across the hierarchy
Method: label-attention weights
Captum is a post-hoc reconstruction; label attention is the model’s own explanation, built into the architecture — each level’s head literally attends to specific words before deciding. Below is that raw attention: one heatmap per head, plus their mean (fixed, always shown side by side, exactly as on Single-level classification), restricted to the 10 classes each head found most plausible (plus the true class, if it happens to fall outside that top 10). Rows are sub-classes-of-that-level, columns are the description’s words. Use the ← / → arrows to switch between examples, and the buttons below the figure to switch between hierarchy levels.
Are these explanations faithful across the hierarchy?
Method: Captum (Integrated Gradients), word-deletion stress test
The same faithfulness test used on the single-level page — delete the words Captum ranked as most important and watch the predicted class’s probability collapse, against a random-removal control — run independently at each level, one small panel per level.
Going Further
The Architecture section flagged this from the start, and Effect of independent heads confirmed it empirically: asking a coarser head directly beats truncating the sub-class model’s own wrong guess by a wide margin — +8.6pp at section / +9.6pp at division for FastText, +9.9pp / +13.1pp for Label Attention — and mistakes really do stay localised to one head rather than dragging every level down together. That is a genuine win. But it is an aggregate statistic, measured over 50,000 test examples. For any single description, nothing in this architecture forces the five heads to agree with one another: they are five independent softmaxes, each trained against its own ground truth, with no term anywhere in the joint loss that ever compares one head’s prediction to another’s:
ℒ = ℒsec + ℒdiv + ℒgrp + ℒcls + ℒsub
five independent cross-entropies, summed — none of them ever look at each other
That summation is also naive in a second way, independent of the consistency question below: it weights every level’s loss equally, regardless of how many classes it has to distinguish (20 at section vs. 679 at sub-class) or how hard that distinction actually is. A fixed heuristic (e.g. weighting by log-class-count) or weights learned jointly with the model (uncertainty-based task weighting, GradNorm-style gradient balancing) are both straightforward extensions to try instead of the plain unweighted sum shown above.
It is entirely possible — and does happen — for the section head to predict “C” on the very same example where the sub-class head’s own true parent is “A”. For a coding officer, five independent guesses that can openly contradict each other is not obviously more useful than the single flat prediction Single-level classification produces: the ambiguity a single sub-class code hides hasn’t disappeared here, it has just moved from “one wrong code” to “five codes that do not even agree with each other on the same activity”. Sharing one token encoder, trained jointly, was hoped to nudge the heads toward implicit agreement — the geometry and word-agreement analyses in Explainability suggest it does so only partially, and only as a side-effect, not by design. Three concrete ways to make the heads actually communicate follow, each trading off a different cost.
A Consistency Term in the Loss
The idea. Add a sixth term to the training loss, alongside the five per-level cross-entropies, that explicitly penalises disagreement between levels instead of hoping the shared trunk produces it as a by-product. For every adjacent pair of levels, compare the coarser head’s predicted distribution against the coarser-level parent of the finer head’s own top prediction — since taking an argmax is not differentiable, this has to be a soft comparison over probabilities, not a hard equality check:
Only adjacent pairs need comparing — section against division, division against group, group against class, class against sub-class — not every pair of levels. Consistency composes along the chain: if each adjacent link agrees, section and sub-class agree automatically as a consequence, with no need for a separate term jumping straight from one end of the hierarchy to the other.
Four adjacent-pair checks, one example
Predicted codes shown below each level — the model's five heads, not the ground truth
↔
✗ parent-of(“62”) = J
↔
✓ agree
↔
✓ agree
↔
✓ agree
Only the Section ↔ Division link disagrees — the chain from Division down to Sub-class is internally consistent.
New term — sum over adjacent pairs only
ℒcons = ∑k CE( P(Lk),
parent-of(argmax P(Lk+1)) )
L1..5 = Section, Division, Group, Class, Sub-class
ℒ = ℒsec+ℒdiv+ℒgrp+ℒcls+ℒsub + λ·ℒcons
Advantages
- Minimal architectural change: still five independent heads, same shared trunk, same inference cost — the fix lives entirely in the training loss, not in the model.
- Directly targets the actual failure mode this page measured (disagreement between levels on the same example), rather than hoping raw accuracy gains fix it as a side-effect.
- A single scalar λ controls the trade-off against the five per-level cross-entropies, and can be tuned — or annealed over training — independently of everything else.
Disadvantages
- Only a soft constraint: it makes disagreement less likely on average, it does not make it impossible — two heads can still contradict each other on any given example at inference time.
- Making it differentiable is genuinely fiddly. As written above,
parent-of(argmax P(L_{k+1}))) runs an argmax — a discrete, non-differentiable operation — so in practice it has to be treated as a fixed pseudo-label at each step (a .detach(), no different from an ordinary ground-truth target). That makes the term one-directional: gradients flow into the coarser head (pushing it to match the finer head’s current guess), but not back into the finer head, which never learns anything from this term. A “clean” fix exists — replace the argmax with the finer head’s full distribution, marginalised into a soft target over parent classes via a fixed class→parent membership matrix (P(L_{k+1}) @ M, then CE(P(L_k), that) or a KL-divergence), which is differentiable both ways — but it means building and maintaining that membership matrix for every adjacent pair of levels (up to 679 × 20 entries at the coarsest split), for a bidirectionality that may or may not matter much in practice.
Cascaded Heads: Coarse → Fine
The idea. Stop feeding every SentenceEmbedder the exact same shared trunk output in parallel — the fan-out shown in the Architecture diagram above. Instead, chain the five levels: the section head’s own sentence embedding, taken before its final linear layer, is concatenated onto the shared trunk output before it ever reaches the division head’s SentenceEmbedder, and so on down to sub-class. Each level literally conditions on what the coarser level just decided, mirroring how a human coding officer actually works — settle on the section first, then narrow down.
Cascade — same SentenceEmbedder → Linear → ŷ per level, chained instead of forked
Section
Division
Group
Class
Sub-class
SentenceEmbedder
input: T
→esec
SentenceEmbedder
input: T⊕esec
→ediv
SentenceEmbedder
input: T⊕ediv
→egrp
SentenceEmbedder
input: T⊕egrp
→ecls
SentenceEmbedder
input: T⊕ecls
↓ esec
↓ ediv
↓ egrp
↓ ecls
↓ esub
↓
↓
↓
↓
↓
ℒsec
ℒdiv
ℒgrp
ℒcls
ℒsub
⊕ = concatenation. Every column keeps the exact same SentenceEmbedder → Linear → ŷk → ℒk shape from the Architecture diagram above — the only change is what feeds each SentenceEmbedder: the shared trunk T alone for Section, T concatenated with the previous level's own embedding for every level after it.
Advantages
- Consistency by construction, not by training incentive: a finer head structurally cannot ignore the coarser levels’ representations, since they are part of its own input.
- Mirrors the coding-officer workflow this page’s own framing cares about: section first, then narrow down, rather than five simultaneous, unrelated guesses.
Disadvantages
- Purely sequential: five heads deep, computed one after another instead of in parallel off one shared trunk — slower to train and slower at inference, at exactly the scale (1.2M training examples) the Hyperparameter sweep above already found expensive to iterate on.
- Errors now propagate forward: a wrong section early in the chain can drag every downstream head down with it, in a way today’s independent heads structurally cannot.
Cross-Level Attention
The idea. Keep all five levels’ SentenceEmbedder outputs exactly as they are today, but insert a small attention block between them and their linear heads: treat the five per-level sentence embeddings as a sequence of five “tokens” and let each one attend to the other four before classification. A linear-attention formulation, rather than plain softmax self-attention, keeps this cheap — though with only five tokens the quadratic cost of full attention is already trivial, so the real appeal of a linear formulation here is a simpler, lower-variance mixing rule, not asymptotic complexity. What the attention block hands off to the five linear heads is then a genuine design choice, with two natural options:
- Variant A — stay separate. Each level keeps its own enriched embedding (e′sec, …, e′sub), now informed by the other four, and its own unchanged linear head reads only its own embedding — closest to today’s architecture, attention is the only new component.
- Variant B — pool into one. The five enriched embeddings are pooled (mean, or a learned weighted pool) into a single shared embedding, and all five linear heads read that same vector. The only thing that can still make the five heads disagree is their own weights — the representation they see is, by construction, identical.
Five per-level embeddings, attending to each other — then two ways to classify
⇅⇅⇅⇅⇅
Linear attention
5 levels attend to each other, bidirectionally
↙
either / or — not both
↘
Variant A — stay separate
e′sec
e′div
e′grp
e′cls
e′sub
↓↓↓↓↓
Linear
Linear
Linear
Linear
Linear
↓↓↓↓↓
Each head still reads its own (now enriched) embedding.
Variant B — pool into one
↓
↙ ↓ ↓ ↓ ↘
Linear
Wsec
Linear
Wdiv
Linear
Wgrp
Linear
Wcls
Linear
Wsub
↓↓↓↓↓
Same efused feeds all five heads — only their own weights (Wsec…Wsub) can still make them disagree.
Advantages
- Bidirectional: unlike the cascade above, a fine-level signal can inform a coarse prediction too, not only the reverse — the model learns which levels are informative for each other, rather than a fixed coarse-to-fine order being imposed on it.
- Still one parallel pass over the five levels, not a five-deep sequential chain — much closer to today’s cost profile than the cascade option above.
- Variant B pushes consistency further than Variant A for free: forcing every head to read the exact same fused embedding removes one whole source of disagreement (different inputs) — whatever the heads still disagree on has to come from their own weights, not from seeing different things.
Disadvantages
- More moving parts than either other option: a new attention module (and, for Variant B, a pooling step) sits on top of the shared trunk and the five heads, adding parameters and one more thing to tune.
- Variant B trades away per-level specialisation for that extra consistency: a single pooled embedding is a narrower bottleneck than five separate ones, so whatever nuance a level-specific embedding could capture on its own has to survive being averaged in with the other four first.