Towards Interpretable Text Classification for Administrative Coding: A Study at INSEE
Internship Report · July 2026
1 Introduction
This report presents the work carried out during an internship at INSEE (Institut national de la statistique et des études économiques), France’s national statistics institute.
The internship focuses on text classification, a core natural language processing (NLP) task that consists in assigning a predefined category to a piece of text. At INSEE, a concrete use case is the automatic coding of business creation declarations: when a company is founded, its activity is described in free text by the declarant and must be assigned a code from the NAF (Nomenclature des Activités Françaises), a hierarchical classification of economic activities. A model has recently been deployed to automate this coding process — a classic single-level FastText classifier, predicting the finest-grained NAF code directly — making it a concrete reference point for evaluating the lightweight architectures studied in this internship. Because NAF codes are organised hierarchically (section, division, group, class, sub-class), models can also be trained to predict several of these levels jointly instead of only the finest one — an approach known as multilevel classification.
Large language models (LLMs) achieve strong performance on text classification, but they come with significant computational costs. In a production environment processing millions of records, lighter and faster alternatives are desirable. This internship investigates whether lightweight neural architectures such as FastText-style mean-pooling models and small transformer encoders can reach competitive accuracy at a fraction of the cost. A secondary focus is explainability: in high-uncertainty settings, decision-makers need more than a bare prediction, and post-hoc explanations can provide the additional information required to act with confidence. All experiments are built on top of torchTextClassifiers, a Python package developed at INSEE that provides a unified interface for training and evaluating these architectures.
The main objectives are:
- Benchmark lightweight text classification architectures (FastText, small transformer encoders) across several datasets of increasing difficulty.
- Study the impact of key hyperparameters (embedding dimension, vocabulary size, number of encoder layers, attention heads, learning rate) on accuracy and data efficiency.
- Investigate explainability methods to understand what these models rely on, and assess whether their predictions can be trusted enough to support human decision-making in uncertain cases.
- Study a multilevel classification approach exploiting NAF’s hierarchical structure: predicting coarser levels alongside the target sub-class can provide auxiliary training signal and a fallback prediction (e.g. the correct division) when the model is not confident enough at the finest level.
This report is organised as a small website, one tab per topic: this page covers the introduction and the shared study setup, followed by dedicated tabs for the Amazon, CLINC150, and NAF results, and a Concepts tab gathering in-depth explanations of the mechanisms and explainability methods used throughout.
2 Study Setup
2.1 Datasets
The study covers three datasets of increasing number of classes, designed as a progression towards the target application: automatic NAF coding, which involves around 700 classes. Experiments were run in this order — Amazon Reviews, then CLINC150, then NAF — each step raising the difficulty before tackling the target task.
Amazon Reviews (English) is the Multilingual Amazon Reviews Corpus (MARC). The English split contains 200,000 training and 5,000 test product reviews, each rated from 1 to 5 stars. The input combines the review title and body. The 5-class ordinal nature of the task makes it harder than binary sentiment.
CLINC150 is an intent detection dataset. It contains 150 intent classes covering everyday conversational domains (banking, travel, home automation, etc.), with 100 training examples per class. The large number of classes and the short, colloquial nature of utterances make it a natural intermediate step between Amazon and NAF.
NAF is the target application: automatic coding of business creation declarations against the Nomenclature des Activités Françaises, with around 700 classes. Results on this dataset constitute the core deliverable of the internship.
The class counts and train/val/test splits used in our experiments are summarised below (NAF is detailed further in its own dedicated section).
| Dataset | Classes | Train | Val | Test | SOTA (acc.) |
|---|---|---|---|---|---|
| Amazon Reviews | 5 | 195,000 | 5,000 | 5,000 | 63.3% (mBERT) |
| CLINC150 | 150 | 15,000 | 3,000 | 4,500 | ~97% (large LMs) |
| NAF | ~700 | — | — | — | — |
2.2 Models
Four architectures of increasing complexity are compared, combining two model families with two sequence aggregation strategies:
- FastText: embeddings are averaged across tokens and fed directly to a classification head. No attention mechanism, minimal parameter count, very fast at both training and inference.
- FastText + Label Attention: same backbone as FastText, but the mean pool is replaced by a cross-attention step where one learned embedding per class acts as the query, allowing the model to focus on the tokens most relevant to each candidate class.
- FATE (FastText Attentive Transformer Encoder) : transformer encoder blocks (multi-head self-attention + feed-forward) are stacked on top of the embeddings before mean pooling, capturing contextual interactions between tokens.
- FATE + Label Attention: same transformer backbone, but with label attention as the aggregation step instead of mean pooling.
over seq_len
tokens → keys/vals
per class
→ (B, emb_dim)
tokens → keys/vals
per class
All four architectures use a WordPiece tokenizer trained on the training data, with vocabulary sizes ranging from 2,000 to 20,000 tokens.
2.3 Experimental Setup
All experiments are tracked with MLflow and run on GPU via Argo Workflows on the INSEE datalab infrastructure. For each dataset, a grid search is run over the relevant hyperparameters (embedding dimension, number of layers, vocabulary size, learning rate, attention heads, label attention); the exact grid and number of configurations vary by dataset and are detailed in the corresponding dataset’s “Hyperparameter sweep” section.