Project · NLP · 2025 · solo
Autoregressive decoding strategies
Greedy, beam search, top-k and nucleus sampling compared on an LSTM language model trained on WikiText-2.
Problem
A language model outputs a distribution over the next token; how you pick from it decides whether the text is repetitive, coherent or surprising. The goal was to build the four standard strategies by hand and look at the quality–diversity trade-off each one makes.
Method
A word-level LSTM (300-d embeddings, two layers of 512 units, dropout 0.5) trained for 30 epochs on WikiText-2. Greedy, beam search (width 5 and 10), top-k (k = 5, 10) and nucleus sampling (p = 0.6) were written from scratch, scored with BLEU and ROUGE, and visualised as beam trees and probability cut-offs.
Results
Beam search with width 10 had the highest BLEU, and tied with width 5 for the highest ROUGE-L. Greedy and nucleus (p = 0.6) produced identical continuations, which suggests a tight nucleus collapses to the argmax on a small model.
Measurements
| Setting | Value |
|---|---|
| Greedy | BLEU 0.348 · ROUGE-L 0.546 |
| Top-k (k = 10) | BLEU 0.289 · ROUGE-L 0.539 |
| Top-k (k = 5) | BLEU 0.191 · ROUGE-L 0.364 |
| Nucleus (p = 0.6) | BLEU 0.348 · ROUGE-L 0.546 |
| Beam (width 5) | BLEU 0.349 · ROUGE-L 0.583 |
| Beam (width 10) | BLEU 0.405 · ROUGE-L 0.583 |
Scores on the prompt “Climate change remains one…”, against the reference sentence. Two prompts in total, so read these as an illustration, not a benchmark.
Stack
Python · PyTorch · NLTK · NumPy · Matplotlib · NetworkX