Interpretable Text Classification for Automatic NAF Coding

Internship Defense · Master 1 Informatique, Décision, Données (IDD) · Université Paris-Dauphine – PSL

Julien Bernardi

2026-09-02

Table of Contents

I. The Internship — Subject & Objectives

  • Subject & objectives
  • The NAF nomenclature — a five-level hierarchy

II. Technical Context

  • The existing production system
  • Tools, infrastructure & scaling

III. Contributions & Results

  • Methodology, models & aggregation methods
  • Amazon & CLINC150: the progressive testbed
  • NAF (single-level): results & explainability
  • Towards multi-level classification
    • Architecture
    • Results & the disagreement problem
    • Going further

IV. Technical Conclusion

V. Personal Assessment

The Internship — Subject & Objectives

Subject

  • Subject stated at the outset — and unchanged through completion: study lightweight, interpretable text classification, applied to INSEE’s automatic NAF coding of business-creation declarations
  • Four objectives:
1
Benchmark
  • Lightweight architectures, benchmarked across datasets of increasing difficulty
2
Understand
  • Impact of embedding size, vocabulary, layers, attention heads, data volume
3
Explain
  • Can predictions be trusted enough to support a human decision on ambiguous cases?
4
Exploit hierarchy
  • NAF is a 5-level nomenclature — can predicting several levels jointly help?

The NAF Nomenclature — A Five-Level Hierarchy

Worked example — declared activity: “Clothing retail in a physical store”

Sectionletter A–U20
GWholesale & retail trade
↓
DivisionXX · 2 digits84
47Retail trade
↓
GroupXX.X274
47.7Other retail sale
↓
ClassXX.XX586
47.71Retail sale of clothing
↓
Sub-classXX.XXY679
47.71ZRetail sale of clothing in specialised stores

The section letter isn’t written into the numeric code — it’s derived from it (each division belongs to exactly one section).

  • A company’s activity, declared as free text, must be assigned a NAF code (aligned with NACE Rev. 2.1)
  • One free-text declaration → one sub-class, the finest of the five levels — a 679-way classification problem

Technical Context

The Existing System

  • INSEE already runs a production single-level FastText classifier, predicting the finest sub-class directly from the free-text declaration
  • Raw accuracy isn’t the gap. Two things are missing from it:
    • Explainability — no way to know why the model chose a given label
    • Smarter mistakes — when it does get the sub-class wrong, nothing guarantees it at least got the right section

Existing Tools & Technical Environment

Python 3.13uvPyTorchPyTorch LightningCaptumKubernetes / Argo WorkflowsQuartoSSP Cloud (Onyxia)S3 / MinIO

  • torchTextClassifiers — INSEE’s own open-source PyTorch/Lightning package, a unified interface across every architecture compared
  • All experiments run on SSP Cloud “Onyxia”, INSEE’s Kubernetes-based datalab
▶
Launch a service
One click → a Docker container pre-installed with VS Code + PyTorch
→
⚡
Choose your GPU
None, one, or several GPUs on demand
→
☁
Shared storage
S3 (MinIO) bucket, same data across every service

Scaling & Reporting

⚙
Configure
Hydra YAML configs — one entry point for every dataset and architecture
→
⇄
Train at scale
Argo Workflows on Kubernetes — hundreds of GPU sweeps in parallel
→
◎
Track & explain
MLflow logs every run · Captum explains predictions
→
▤
Report
Quarto site, auto-deployed to GitHub Pages

Contributions & Results

Methodology: A Progressive Testbed

Rather than start directly on NAF’s ~700 classes, the whole pipeline — training, prediction analysis, explainability — was built and validated on two public benchmarks first, in increasing order of difficulty.

1
Amazon Reviews
  • 5-class sentiment (1–5★)
  • ~200,000 reviews
→
2
CLINC150
  • 150-class intent detection
  • ~15,000 utterances
→
3
NAF
  • Target application
  • ~1.2M+ declarations

Models

  • Minimal parameters — very fast to train and to run
  • The baseline behind the model already in production for NAF coding
(B,) str
─────▶
⌨
WordPiece
(B, seq_len)
────▶
int64
⊞
Embedding
vocab_size × emb_dim
(B, seq_len, emb_dim)
────▶
float32
σ
(B, seq_len, emb_dim) · float32
∑÷n
Aggregation Method
see next slide
(B, emb_dim)
────▶
float32
σ
(B, emb_dim)
────▶
float32
⊸
Linear
emb_dim → n_classes
(B, n_classes)
────▶
ŷ (logits)
  • Same embeddings, but first passed through N transformer encoder layers (self-attention + feed-forward)
  • Words gain context from their neighbors before being pooled
  • More expressive representation — but heavier and slower to train/run
(B,) str
─────▶
⌨
WordPiece
(B, seq_len)
────▶
int64
⊞
Embedding
vocab_size × emb_dim
(B, seq_len, emb_dim)
────▶
float32
σ
(B, seq_len, emb_dim) · float32
↓
× n_layers
σ
(B, seq, emb)
────▶
pre-attn
Q K V
Multi-Head Attn
head_dim = emb_dim / n_head
↺ +res
────▶
(B, seq, emb)
σ
(B, seq, emb)
────▶
pre-MLP
⇔
MLP
emb_dim → 4·emb_dim → emb_dim
↺ +res
(B, seq_len, emb_dim) · float32
↓
σ
(B, seq, emb)
────▶
float32
∑÷n
Aggregation Method
see next slide
(B, emb_dim)
────▶
float32
σ
(B, emb_dim)
────▶
float32
⊸
Linear
emb_dim → n_classes
(B, n_classes)
────▶
ŷ (logits)

Aggregation Methods

  • Every token embedding contributes equally to the sentence vector: v = (1/T) · Σt et
  • The same pooled vector v is then reused by the linear head, for every class alike
Input token embeddings
e1
"sells"
e2
"organic"
e3
"vegetables"
e4
"online"
↓
v = (1/T) Σt=1T et
v
pooled sentence vector, v ∈ ℝd
— the same v is passed to every class —
↓
ŷc = wc · v + bc
ŷ1
ŷ2
ŷ3
class scores, C = 3 shown
  • Each class c learns its own attention query, giving weights αc,t over the tokens (softmax, sums to 1)
  • Class vector: vc = Σt αc,t · et — a different weighted combination per class
  • The weights double as an explanation: which words mattered, for which class
Input token embeddings
e1
"sells"
e2
"organic"
e3
"vegetables"
e4
"online"
vc = Σt αc,t et   (one query per class)
class A — αA,t
.05
.62
.20
.13
↓
vA
ŷA = wA·vA + bA
ŷA
class B — αB,t
.04
.09
.17
.70
↓
vB
ŷB = wB·vB + bB
ŷB

Amazon: Hyperparameter Sweep

960 runs on Amazon, swept across architecture and embedding dimension.

Figure 1: Validation loss vs. number of parameters
Figure 2: Effect of embedding dimension on validation loss

Amazon: Predictions

Figure 3: Prediction distance per architecture
  • The vast majority of predictions land within ±1 star of the truth — off-by-3+ errors stay under 8%

Amazon: Explainability

Each word’s background is shaded by its attribution score for the selected class — green = pushes toward it, red = pushes away. For Label Attention’s own weights (third row), blue intensity = share of attention received — unsigned, since attention weights can’t push away from a class, only toward it.

Figure 4: Highlighted-text view combining Mean Pooling’s and Label Attention’s Captum IG scores with Label Attention’s own attention weights.
Label attention — "very disappointed these were nothing like
their terrific potato chips from my childhood."
true: 1★ · Label Attention → 1★ ✓
Figure 5: Label-attention weights for one example review: each of the 4 heads’ (class × word) attention matrix.

CLINC150: Hyperparameter Sweep

Figure 6: Validation loss vs. number of parameters

A clear gap: label attention consistently hurts here, unlike on Amazon. Suspected cause — not enough data per class: 150 classes for 15,000 examples here, vs. 5 classes for 200k on Amazon, so building a good class representation is much harder. We’ll confirm this later.

CLINC150: Explainability

Per-head self-attention — "i need some directions to phoenix"
true: directions · Label Attention → tell_joke ✗ · Mean Pooling → directions ✓
Figure 7: Self-attention matrices per head and per layer, for a sampled CLINC150 utterance.

NAF: The Core of the Project

Amazon and CLINC150 were the testbed. NAF is the target application — automatic coding of business-creation declarations, at production scale.

Label Attention: The Payoff

Figure 8: Validation loss vs. training set size, by architecture
Training examples FastText + Label Attn.
163,615 ≈0.63 ≈0.76
314,230 ≈0.57 ≈0.63
628,461 ≈0.51 ≈0.53
1,208,579 ≈0.48 ≈0.47

This confirms the hypothesis from CLINC150: label attention needs enough data per class to learn useful queries, and only pays off once that data is available — exactly what happens here, at full NAF scale.

NAF: Predictions

Figure 9: Among wrong predictions, the coarsest NAF level at which they still land correctly

Only ≈7–8% of errors stay within the right class — the largest bucket by far, ≈47–48%, is “No match”: missing even the section.

Figure 10: Example predictions on the NAF test set, across all five hierarchy levels (derived by truncating the sub-class prediction)

NAF: Explainability — Faithfulness

Progressively delete the most-attributed words and track how fast the model’s own confidence collapses, against a random-deletion control.

Figure 11: Faithfulness test: retained probability as top-attributed words are deleted (solid = guided, dotted = random control).

Label attention’s own weights are the most faithful guide of the three — a free, more trustworthy explanation at this scale.

NAF: The Real Problem

With a single predicted class, an error is an error.

One model, one flat 679-way sub-class prediction — no notion of section, division, group or class along the way. As just seen: a wrong prediction is roughly as likely to be completely off as it is to be a near-miss, and it’s structural — not fixed by picking a better architecture.

Today — Wrong Is Wrong
  • One flat sub-class prediction
  • No notion of “how close”
  • A miss is a miss, however far off
The Goal — Smart Mistakes
  • Hierarchy-aware predictions
  • Wrong sub-class ≠ wrong section
  • Fail gracefully, not catastrophically

The goal isn’t fewer errors — it’s smarter ones.

Towards Multi-Level Classification

Single-level NAF hits a ceiling on a 679-way problem. Can predicting all 5 levels jointly push past it?

Multi-Level Classification: Architecture

  • Give the model that hierarchy explicitly — one independent head per level, on top of a shared token encoder
  • L = L_sec + L_div + L_grp + L_cls + L_sub — unweighted sum, backpropagated jointly through the shared trunk
  • The bet: learning on every level at once, not just the finest one, might push past the ceiling a lone single-level model hits on a 679-way problem
  • Goal: not necessarily fewer errors overall, but smarter ones — the sub-class head can misfire while section/division heads still land correctly
Input
"Conseil en stratégie et management"
(B,) · str
Shared Token Encoder — weights shared across all levels
⌨
WordPiece
vocab_size
(B, L) · int64
────►
⊞
Embedding
V × d
(B, L, d) · float32
────►
σ
RMSNorm
(B, L, d) · float32
────►
T
Token embeddings
reused by every head
Level 1 · Section
∑÷n
SentenceEmbedder
Mean Pool or Label Attn
(B, d) or (B, C, d)
⊸
Linear
d → 20 | 1×d
ŷ̂1
"A"
20 classes
ℒsec
· · ·
Level k · …
∑÷n
SentenceEmbedder
Mean Pool or Label Attn
(B, d) or (B, C, d)
⊸
Linear
d → Ck
ŷ̂k
Ck classes
ℒk
· · ·
Level 5 · Sub-class
∑÷n
SentenceEmbedder
Mean Pool or Label Attn
(B, d) or (B, C, d)
⊸
Linear
d → 679 | 1×d
ŷ̂N
"7010Y"
679 classes
ℒsub
Joint training loss
ℒ = ℒsec + ℒdiv + ℒgrp + ℒcls + ℒsub
Independent cross-entropy at each level — summed and back-propagated through the shared trunk

Class counts (20 / 84 / 274 / 586 / 679) reflect codes observed in the training split (train_fraction = 0.13 of the full corpus). Full NAF 2025: 21 / 87 / 287 / 651 / 747.

Multi-Level: Results

Figure 12: Among wrong sub-class predictions, the finest level that still lands correctly — multi-level’s independent heads vs. single-level’s truncated cascade

We reason directly with each head’s own prediction, independently — not by truncating the sub-class guess.

Figure 13: Example predictions across all five hierarchy levels

The heads are independent: a fine level can land right while a coarser one misses — impossible before.

Multi-Level: A New Problem Appears

  • The 5 heads are trained jointly but never compared to each other at inference — nothing in the loss enforces agreement
  • Consequence: heads can produce mutually inconsistent codes — e.g. the section head says “C” while the true parent of the predicted sub-class is section “A”

As is, that’s not useful for a human annotator — but a smarter, consistency-aware model could at least hand them better guidance.

Multi-Level: Going Further

Intuition: add a sixth loss term that directly penalizes disagreement between adjacent levels.

Four adjacent-pair checks, one example

Predicted codes shown below each level — the model's five heads, not the ground truth

Section
"M"
↔ ✗ parent-of(“62”) = J
Division
"62"
↔ ✓ agree
Group
"620"
↔ ✓ agree
Class
"6201"
↔ ✓ agree
Sub-class
"6201Y"
New term — sum over adjacent pairs only
ℒcons = ∑k CE( P(Lk), parent-of(argmax P(Lk+1)) )
L1..5 = Section, Division, Group, Class, Sub-class
ℒ = ℒsec+ℒdiv+ℒgrp+ℒcls+ℒsub  +  λ·ℒcons
Pros
  • Minimal change — only the loss changes
  • Targets the exact failure mode measured
  • One scalar λ tunes the trade-off, can be annealed over training
Cons
  • Only a soft constraint — disagreement less likely, never impossible
  • The clean differentiable version needs a full class→parent membership matrix per adjacent pair

Intuition: chain the levels instead of feeding them all in parallel — each head's embedding feeds the next, section first, mirroring a human coding officer.

Cascade — same SentenceEmbedder → Linear → ŷ per level, chained instead of forked

Section
Division
Group
Class
Sub-class
SentenceEmbedder
input: T
→esec
SentenceEmbedder
input: T⊕esec
→ediv
SentenceEmbedder
input: T⊕ediv
→egrp
SentenceEmbedder
input: T⊕ecls
→ecls
SentenceEmbedder
input: T⊕ecls
↓ esec
↓ ediv
↓ egrp
↓ ecls
↓ esub
Linear
Linear
Linear
Linear
Linear
↓
↓
↓
↓
↓
ŷsec
ŷdiv
ŷgrp
ŷcls
ŷsub
ℒsec
ℒdiv
ℒgrp
ℒcls
ℒsub

⊕ = concatenation. Every column keeps the exact same SentenceEmbedder → Linear → ŷk → ℒk shape — the only change is what feeds each SentenceEmbedder: the shared trunk T alone for Section, T concatenated with the previous level's own embedding for every level after it.

Pros
  • Consistency by construction — a finer head structurally cannot ignore the coarser levels
  • Mirrors the coding-officer workflow: section first, then narrow down
Cons
  • Purely sequential — 5 heads deep, slower to train and to run than parallel heads
  • Errors propagate forward — a wrong section can drag every downstream head down with it

Intuition: keep all 5 heads in parallel, but let their embeddings communicate before classification — mixed by a single function (mean, attention…)

Five per-level embeddings, averaged into one before classification

Section
esec
Division
ediv
Group
egrp
Class
ecls
Sub-class
esub
↘↓↓↓↙
mean
↓
efused
↙ ↓ ↓ ↓ ↘
Linear
Wsec
Linear
Wdiv
Linear
Wgrp
Linear
Wcls
Linear
Wsub
↓↓↓↓↓
ŷsec
ŷdiv
ŷgrp
ŷcls
ŷsub

Same efused feeds all five heads — only their own weights (Wsec…Wsub) can still make them disagree.

Pros
  • Removes one whole source of disagreement for free: identical input to every head
  • Still one parallel pass — much closer to today's cost than the cascade
  • No extra module to train — the mean is parameter-free
Cons
  • Trades away per-level specialisation: fine-grained signal gets diluted into the average
  • Every head reasons from the same coarse summary, not from what makes its own level distinctive

Technical Conclusion

Answering the Internship’s Objectives

1. Accuracy vs. cost — Lightweight architectures are competitive, and label attention’s extra expressiveness is real — but it only pays off once there is enough data per class. Confirmed twice: CLINC150 vs. Amazon, then again on NAF at 1.2M-example scale.

2. Trustworthy explanations — Through extensive testing and illustration, we succeeded in identifying what drove the model to give a given answer.

3. Hierarchy — Dedicated per-level heads cut the “completely wrong” rate from ~47% to 29–34%, and clearly beat truncating a flat prediction — but introduce a head-disagreement problem that this internship identifies without yet solving.

Personal Assessment

Skills Developed

  • End-to-end ML engineering at production scale: PyTorch Lightning, Hydra-based configuration, MLflow experiment tracking, GPU sweeps orchestrated with Kubernetes / Argo Workflows
  • Explainability in practice: Captum Integrated Gradients, attention-based interpretability, and how to actually measure whether an explanation is faithful rather than just plausible-looking
  • Scientific communication: structuring three months of experiments into a public, reproducible Quarto report and a reusable, config-driven codebase
  • Working with a real, production dataset: this is not a clean academic benchmark — it actually feeds a production system at INSEE, which changes everything about how the work has to be done
  • Team communication & collaboration: discussing and iterating with the team, and especially collaborating on GitHub — branches, pull requests, code review — where I learned a lot about the field’s best practices

Thank You

Questions?

Full report, code, and figures on my GitHub: github.com/julber95/interpretable-text-classification