- A language model does one thing over and over: given the text so far, it gives every word in its vocabulary a probability of coming next, then one word is picked and the loop repeats.
- The simplest way to get those probabilities is to count, in a body of text, which word followed which. Counting works until the contexts get long enough that almost none of them have been seen before.
- Temperature, top-k, and top-p don't change what the model knows. They change how a word is picked from its probabilities.
- A neural network replaces the count table with arithmetic on word vectors, so words used in similar places share what was learned about them, and the model can score contexts it never saw.
Start typing a message on a phone and three words appear above the keyboard. Tap one, and three more appear. The phone is guessing the next word, and it's often right. The program that makes those guesses is a language model, and so, at a much larger scale, is the program behind a chatbot. Both do the same small thing over and over: look at the text so far and give every word they know a probability of coming next.
Every model on this page is real, and small enough to watch. They learn from a 92-sentence story about a cat, a dog, a bird, a boy and a girl, shipped inside this page; every number in every figure is computed from it as you move things. Here's the first of them, a phone keyboard that suggests a next word. Tap a suggestion, or press the button, and watch the probabilities on the right change with the last word:
After “the” it offers cat, dog, and girl; after “cat”, sat, a full stop, and drank. Keep tapping the middle word and you get grammatical nonsense, something like “the cat sat on the cat sat on the…”. Every model on this page, and every chatbot, writes text this way, one piece at a time. What differs is how the probabilities are computed and how a word is picked from them. This page builds up both, from counting to a small neural network, and stops just short of the trick that makes modern models work, which gets its own page.
From text to tokens
A model can't read letters. It works with numbers, so the first job is to cut text into pieces and give each piece a number. The pieces are called tokens, and the list of every token a model knows is its vocabulary.
The simplest choice is one token per word, plus one for the full stop. Our story uses 69 different words, so its vocabulary has 70 tokens, numbered from 0 in order of how often they appear. Pick a sentence and see its tokens and their ids:
The model will only ever see the numbers. “the cat drank the milk.” arrives as 0, 2, 16, 0, 25, 1, and whatever comes out is a number too, turned back into a word only at the very end. Nothing about the number 2 says “small furry animal”. Whatever the model comes to know about cats, it has to learn from where 2 appears.
Whole words have a problem, though. A word that isn't in the vocabulary can't be read at all, and real text is full of names, typos, and new words. The opposite choice, one token per letter, can read anything, but then even a short sentence is a long list, and the model has to learn spelling before it can learn anything else.
Most real models split the difference with byte-pair encodingThe method began as a way to compress files ; Sennrich, Haddow and Birch turned it into a way to cut words into pieces for translation .. Start with single characters. Find the two symbols that sit next to each other most often in the text, glue them into one new token, and repeat. Common words end up as one token, rare words as a few pieces, and anything at all can still be spelled out letter by letter. Step through the learned from our story:
With no merges the two sentences are 50 tokens, one per character, and the vocabulary is just the 23 characters the story uses. The first merges are the commonest pairs in English-like text: h and e, then a space and t, then those two together. By merge 3 the whole word “·the” (with its leading space) is a single token. After 90 merges almost every word in our story is one token, and the sentences are 16 tokens long. A word the story never used, like “dragon”, would still come out as a handful of smaller pieces, so nothing is ever unreadable.
Each merge is chosen by counting. Here are the ten commonest neighboring pairs at the step you choose; the top one becomes the next merge:
Real tokenizers do exactly this on far more text, usually on raw bytes rather than letters, and stop after tens of thousands of merges. GPT-2, for example, has a vocabulary of 50,257 tokens . For the rest of this page we go back to one token per word: with only 70 of them, every table and every probability fits on the screen.
Byte-pair encoding in fifteen lines 2 min
This is the code the figures above run, trimmed. pieceFreq counts each word-with-its-space in the story; each round counts every neighboring pair, weighted by how often its word appears, and merges the winner everywhere.
const merges = [];
let split = new Map([...pieceFreq.keys()].map(p => [p, [...p]]));
for (let m = 0; m < 90; m++) {
const pairs = new Map();
for (const [piece, symbols] of split)
for (let i = 0; i + 1 < symbols.length; i++) {
const key = symbols[i] + "\u0000" + symbols[i + 1];
pairs.set(key, (pairs.get(key) || 0) + pieceFreq.get(piece));
}
const [best] = [...pairs].sort((a, b) => b[1] - a[1]);
const [a, b] = best[0].split("\u0000");
merges.push([a, b]);
for (const [piece, symbols] of split) split.set(piece, mergeOnce(symbols, a, b));
}To cut new text, split it into words the same way and apply the merges in the order they were learned. Ties are broken alphabetically here, so the result is the same on every load.
Counting what comes next
Now the model itself. The simplest one that works is a table: for every word, count which words came right after it in the story. Choose a word and see what followed it:
After “the” comes a long tail of nouns: cat, dog, girl, boy, and so on down to words that appeared once. After “drank” comes “the” every time, because in this story you only ever drink the something. After “was” come mostly describing words: cold, happy, warm, tired. The table already knows a little grammar, though nobody told it any: it has only counted.
Put the counts for the commonest words side by side and you get a grid, with the previous word down the side and the next word across the top. Point at a cell to read it, and switch on probabilities to divide each row by its total:
That's a complete language model, called a bigram model: two words, one predicting the other. To use it, look up the row for the last word and read off the probabilities. After “the”, cat gets 17%, dog 12%, girl 9%. The grid is mostly empty; that's normal. Of the 4,900 possible pairs of our 70 tokens, the story uses 193.
Written as a formula, the probability that word b follows word a is the count of the pair divided by the count of everything that followed a:
Read the bar \mid as “given”. Everything else on this page is about computing that left-hand side better.
Picking a word
A row of probabilities doesn't say which word to write. Something has to choose. The keyboard let you choose; a model writing on its own has to do it by itself, and the usual way is to draw at random, with each word's chance of being drawn equal to its probability.
Here's how a computer does that. Lay the probabilities end to end along a strip from 0 to 1, so each word gets a stretch as long as its probability. Then pick a random number between 0 and 1 and see whose stretch it lands in. Drag the marker, or set it to :
Big stretches catch the marker often and small ones rarely, in exact proportion to their length. That's all sampling is. Run it again and again, adding each drawn word to the text and moving the context along, and the model writes. Press play, or step one word at a time:
The result reads like a dream of the story. The first seed writes “the girl drank the big dog saw the cat liked the bird.” Each pair of neighboring words is something the story contains, but nothing holds a sentence together beyond that. The model can't remember that a sentence began with “the girl” once it has moved past it, because it only ever looks one word back.
Suppose we let the model look three words back instead of one. What happens to what it writes?
With three words of context, most contexts appeared exactly once in the story, so there's only one word that can come next: the one that came next in the story. The model ends up reciting.
Longer memories, emptier tables
The obvious fix is to look further back: count what followed each pair of words, or each run of three. Models like this are called n-gram models, and for decades they were how speech recognizers and phone keyboards worked. Choose how many words the model looks back and generate a passage; stretches copied word for word from the story, six words or longer, are marked:
Looking two words back, the model writes long runs of the story verbatim, whole sentences one after another; three words back is the same. Switch to and the copied stretches shrink to a few words. That's what the prediction was getting at: with three words of context, most contexts were seen exactly once, so they have exactly one continuation, and the model has nothing to do but recite. A longer memory made it better at the text it learned from and no better at anything new.
The deeper problem is arithmetic. With 70 tokens there are 70 one-word contexts, 4,900 two-word contexts, 343,000 three-word contexts, and the numbers multiply by 70 for every word you add. Our story is 519 words long. Step through the context length and compare how many contexts are possible with how many the story actually contains:
By three words of context, the story covers about one in a thousand of the possible contexts, and 222 of the 320 it does contain appear only once. Every unseen context is a dead end: the table has no row for it, so the model can't say anything at all. A bigger body of text helps less than you'd hope, because the number of possible contexts grows far faster than any library. This is sometimes called the curse of dimensionality, and Shannon ran into it in 1948, generating text from letter and word counts by hand .
Practical n-gram systems patched this with tricks: give unseen pairs a small made-up count (smoothing), or fall back to a shorter context when the long one is missing. They work, up to a point. The lasting fix came from replacing the table with something that can generalize, and before getting to it, it helps to look at how probabilities are produced in a form that makes that possible.
Scores and temperature
A neural network doesn't count. It computes a score for every word in the vocabulary, any number at all, positive or negative, with higher meaning “more likely next”. Scores aren't probabilities: they don't add up to 1 and can be negative. The step that turns them into probabilities is called softmax: raise e to the power of each score, which makes everything positive and stretches the gaps, then divide by the total so they add up to 1.
Here \green{z_i} is word i's score and \red{T} is the temperature, a knob most text generators expose. The scores below come from the small neural network we'll build in a moment; the probabilities are softmax at the temperature you choose.
Turn the temperature right down toward 0. What does the model do?
Dividing by a tiny T multiplies every gap between scores, so after softmax the top word gets nearly all the probability. The draw stops being random.
Try a temperature of , then compare with :
At a temperature of 1 you get the probabilities the model was trained to give. After “sat on”, “the” gets 96%, because in the story you only ever sit on the something. After “the girl” the scores are close together and the top word, “drank”, gets only 11%. Turn the temperature down and the top word takes over in both. Turn it up and the probabilities flatten toward the 1-in-70 of a model that knows nothing.
The readout's spread measures that flatness. It's the entropy of the probabilities in bits: 0 when one word is certain, and log₂ 70, about 6.1 bits, when all 70 are equally likely. Low temperature writes safe, repetitive text; high temperature writes surprising text that stops making sense. Neither changes what the model knows: the scores stay exactly where they were.
Choosing among the likely
Pure sampling has a flaw that matters more for big models than for ours. With tens of thousands of tokens in the vocabulary, the long tail of unlikely ones adds up: each has a tiny probability, but together they can hold a real share, and drawing even one of them can derail a sentence. Two simple rules cut the tail off before drawing:
- top-k sampling keeps only the k likeliest words ;
- top-p sampling, or nucleus sampling, keeps the fewest words whose probabilities add up to at least p .
Either way the kept words' probabilities are scaled up to add to 1 again, and the draw happens among them. Greedy decoding is the extreme: always take the single top word. Compare the rules after “the girl”:
Top-k always keeps the same number of words, whether the model is sure or not. Top-p adapts to how sure the model is. After “sat on”, where “the” has 96%, a nucleus of 0.9 keeps a single word. After “the girl”, where the probability is spread thin, the same 0.9 needs 19 words, and even a nucleus of needs several. That adaptiveness is why nucleus sampling became a common default.
Greedy decoding looks like the safest rule, since it always takes the likeliest word. It has a failure that's easy to see. Generate 30 words greedily and 30 by sampling, from the same start:
The greedy text quickly settles into “the girl drank the water.” over and over. Our network looks back two words, so as soon as it's in a pair of words it has been in before, everything after must repeat exactly. Large models look much further back, but greedy decoding still tends to repeat phrases, and loops like this were one of the problems that motivated nucleus sampling . Sampling avoids the trap by occasionally taking a less likely word.
A network instead of a table
Back to the dead ends. The story has “the cat drank”, “the girl drank”, and “the boy drank”, but never “the dog drank”. A table that looks two words back has no row for “dog drank”, so it can't predict anything after it. Yet you know what comes next: the, followed by something to drink. You know because dogs are like cats in all sorts of ways the story does show: they sit, sleep, eat, chase, and get fed.
In 2003, Yoshua Bengio and colleagues built a model that could make the same leap . Instead of a row per context, it gives every word a short list of numbers, a word vector, and computes the next word's scores from the vectors of the words before it. Words that are used in similar places end up with similar vectors, so whatever the model learns about one carries over to the other. Here's ours, trained on the story, next to the two-word count table. Choose a context the story contains or one it never saw:
After “dog drank”, which it never saw, the network puts 93% on “the”. After “mouse chased”, also unseen, it again says “the”. After “his cat”, it spreads its bets over things cats do in the story: slept, sat, ate, drank. None of that was looked up. It came out of arithmetic on vectors that the network shaped during training. Here's that arithmetic for any two words you choose:
Reading left to right: the two words' vectors, 8 numbers each, are laid end to end into 16 inputs. Each of the 16 hidden units multiplies every input by a weight of its own, adds them up, and squashes the total into the range −1 to 1. Then every word in the vocabulary gets a score: another weighted sum, this time of the hidden units. Softmax turns the 70 scores into probabilities. Change “the dog” to “the cat” and the hidden units change only a little, because the two words' vectors are similar, which is exactly why what the network learned about one carries over to the other.
All of the knowledge is in the numbers: the 70 word vectors, the weights into the hidden units, and the weights out of them, 2,022 numbers in all. None of them was set by hand. They started random, and training adjusted them. The figure below trains a fresh copy from scratch as soon as it's on screen; press reset to watch again. The dark curve is how surprised the model is by the training story; the gold curve is how surprised it is by nine sentences it never trains on:
At step 0 the model is a random guesser, about 6 bits per word, as surprised as you'd be picking blindly among 70 words. Within a couple of hundred steps both curves drop below the counting model's held-out score. The bars on the right sharpen as you watch: “the cat” stops predicting random words and starts predicting a full stop, “ate”, “slept”, “drank”.
Then the two curves part. The training curve keeps falling while the held-out curve bottoms out near step 250 and starts to rise: the network is learning the training sentences themselves rather than what they have in common. That's overfitting, the same disease the three-word count table had, in milder form. Switch on the weight penalty, which pulls every weight a little toward zero at each step so that only patterns that help across many examples survive, and the held-out curve stays down. Try a too: training gets faster and much noisier.
Surprise, cross-entropy, and how the weights get nudged 4 min
The curves measure cross-entropy. When the actual next word had probability p under the model, the model's surprise is -\log_2 p bits: 0 bits if it was certain, 1 bit if it gave 50%, 6.1 bits if it gave 1 in 70. Averaged over every word of a text of N tokens:
Shannon used the same measure to estimate how predictable English is, by asking people to guess text one letter at a time . A model with a cross-entropy of H bits is as uncertain, on average, as if it were choosing uniformly among 2^H words; that number is called the perplexity. Our trained network's 2.9 bits on held-out sentences is a perplexity of about 7.
Training lowers H by gradient descent. For one example, the network computes
and calculus gives, for every weight, how much the surprise would change if that weight moved a little. Softmax makes the first of these unusually simple: the change in surprise per unit of word k's score is \purple{p_k} - 1 for the right word and \purple{p_k} for every other, so training pushes the right word's score up and every other score down, each in proportion to how much probability it took. The chain rule carries that back through the hidden units to the word vectors. Each step averages these over 32 examples and moves every weight a little against its gradient, with the step sizes adapted per weight by the Adam method .
What the vectors learned
The word vectors are 8 numbers each, which is 8 dimensions: too many to draw. But we can find the two directions in which the 70 vectors differ most and flatten the rest away, a standard trick called principal component analysis. Here are the story's nouns, verbs, and describing words on that flat map, colored by the role each plays. Point at a word to see its nearest neighbors in the full 8 dimensions:
The network was never told what an animal is. Yet in all 8 dimensions “dog” is closest to “cat” and “mouse”, “boy” to “girl”, and the things you put after “the”, like milk, mat, roof, and garden, crowd together. Each of those groups fills the same slots in sentences, and filling the same slots is the only thing the training could see. Not every neighbor makes sense on a map this small: some verbs land near unrelated nouns, and with 70 words and 8 numbers each, there isn't room for much more structure. Real models use vectors of thousands of numbers, and the groupings in them are much richer .
How much did all this buy? The measure is the same cross-entropy as in the training figure, now for every model on this page, on the training story and on the nine held-out sentences. Counting models need a little smoothing on unseen text; switch it off to see why:
On the training story, counting three words back is the champion, at 0.6 bits per word: it has nearly memorized the text. On the held-out sentences the order flips. Without smoothing, every counting model hits a pair it never saw and gives it probability zero, which is infinite surprise. With smoothing, counting one word back does best of the counters, and longer contexts do worse, because their tables are emptier. The network beats them all on new text, at 2.9 bits per word, with 2,022 numbers and no table at all. That gap, between memorizing a text and capturing what's general in it, is the whole reason language models are built from networks.
Looking further back
Our network still looks only two words back, and the next word often depends on something much earlier. Take a sentence that isn't in the story: “the keys that the man found in the old box …”. Is the next word is or are? It depends on keys, eight words back. Slide the window of words a model can see:
You could widen the network's input to ten words, laid end to end like our two. But the input grows with every word, and a fixed-width window still treats position 9 as completely separate from position 8, so the network has to learn the same thing separately at every distance. For years, language models used recurrent networks that read text one word at a time and carried a running summary forward, which helped but made long-range connections hard to learn.
The design behind today's models, the transformer , does something different: at every position it lets the model look back over all the earlier tokens at once and decide, from their vectors, which ones matter for what comes next. In our sentence, it can learn to look straight at “keys” and skip the clause in between. That mechanism is called attention, and it has a page of its own.
From toy to real
Everything on this page runs inside a browser tab in under a second. The models behind modern chatbots do the same thing at a scale that's hard to picture. The loop is identical: tokens in, a score for every token in the vocabulary out, softmax, a draw, repeat. What changes is the size of everything in between.
| This page | Large models | |
|---|---|---|
| Vocabulary | 70 words | tens of thousands to a few hundred thousand tokens |
| Context | 2 words | thousands to hundreds of thousands of tokens |
| Numbers learned | 2,022 | billions to hundreds of billions |
| Training text | 519 words | trillions of tokens |
| Vector per token | 8 numbers | thousands of numbers |
Scale changes what the probabilities capture. A model trained on a story about a cat learns that cats drink milk. A model trained on a large share of the written world learns grammar, facts, styles of argument, and the patterns of code, because getting the next token right, across that much text, requires all of them. But it still only does what this page does: give every token a probability and pick one. When people say a model “knows” or “thinks” something, the precise version is that its training made certain continuations very probable.
That framing explains some of the behavior people find surprising. A model can write a confident sentence that's false, because a fluent, confident sentence was a probable continuation. The same question can get different answers, because the answer is drawn, not looked up, and a lower temperature makes the answers more consistent without making them more correct. And it can handle a question nobody has ever asked in exactly those words, for the same reason our little network could finish “the dog drank”.
The 92-sentence story, the tokenizer, both kinds of model, and every number in the figures are computed in this page, in plain JavaScript, when it loads. The neural network is trained for 800 steps from a fixed random seed, so the figures show the same model on every visit.
References
- C. E. Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal 27 (1948), 379–423 and 623–656.
- C. E. Shannon, “Prediction and Entropy of Printed English”, Bell System Technical Journal 30 (1951), 50–64.
- P. Gage, “A New Algorithm for Data Compression”, The C Users Journal 12, no. 2 (1994), 23–38.
- R. Sennrich, B. Haddow, and A. Birch, “Neural Machine Translation of Rare Words with Subword Units”, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (2016), 1715–1725.
- Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A Neural Probabilistic Language Model”, Journal of Machine Learning Research 3 (2003), 1137–1155.
- D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization”, International Conference on Learning Representations (2015).
- A. Fan, M. Lewis, and Y. Dauphin, “Hierarchical Neural Story Generation”, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (2018).
- A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The Curious Case of Neural Text Degeneration”, International Conference on Learning Representations (2020).
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention Is All You Need”, Advances in Neural Information Processing Systems 30 (2017).
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners”, OpenAI technical report (2019).