Attention Is All You Need

The box translates our sentence without recurrence. To see why anyone would want that, look at what recurrence was costing.

A recurrent network reads position by position: the hidden state h_t is computed from h_t−1 and the current word. That little subscript is the whole problem. Each step needs the one before it, so nothing can run in parallel within a sentence — no matter how many processors you own, a 50-word sentence is 50 sequential steps. Memory limits make it worse: you can't just batch your way out at longer lengths.

There's a second cost, quieter but just as real. Watch “its” try to reach “Law” in our sentence:

The paper turns this picture into a table (its Table 1), scoring each layer type on two things we now care about: how many steps must run in sequence, and how long the worst-case path is between two positions. Convolutional stacks (ByteNet, ConvS2S) had already fixed the parallelism — but their path between distant words still grows with distance, linearly or logarithmically. Self-attention is the row where both columns hit bottom:

So the bet is: attention only, everything reachable in one step, everything computable in parallel. Here's the machine that bet produced — outline first, details owed.

The Transformer keeps the classic two-part shape of translation models. An encoder reads all 21 tokens of our sentence at once and turns them into 21 context-loaded vectors. A decoder then writes the German, one word at a time, consulting those vectors as it goes. Each side is a stack of N = 6 identical layers, and every layer contains only two kinds of machinery: an attention sublayer (words exchanging information) and a small feed-forward network (each word thinking by itself). Every vector everywhere has the same width, d_model = 512, so the parts snap together.

Two IOUs before we dive. That “attention sublayer” is really multi-head attention — a phrase that will only make sense once you've computed attention yourself, so Steps 4–7 pay it off. And the decoder's “one word at a time” hides a masking trick, paid off in Step 10. Everything else in the drawing you've already met.

All between-word traffic rides on attention — so let's watch it carry some. No numbers yet; first, what it feels like.

When the authors opened up their trained encoder, they found a head in layer 5 doing something you can read off the plot: at the word “its”, the attention lands sharply on “Law” and “application” — the two words that resolve what the pronoun refers to. That's the paper's Figure 4, and here it is redrawn:

The mechanism behind those arcs runs on three vectors per word. Take the embedding of “its” and multiply it by three learned matrices — W^Q, W^K, W^V — to get a query (the question this word is asking), a key (the label it shows when other words ask), and a value (the content it hands over if selected). Attention is then a soft lookup: the query of “its” is compared against every word's key, and the answer comes back as a weighted blend of every word's value. Not one winner — a blend, weighted by how well each key matches.

If the three-way split feels arbitrary, you're in good company — this is the point where most readers stall, and the paper itself introduces q, k, v in a single sentence and moves on. It helps to notice the roles are genuinely different: when “its” looks for its noun, the question it asks (“I'm a possessive pronoun seeking an owner”) is not the label it shows others (“I'm a determiner-ish little word”), and neither is the content it would contribute. One vector can't play all three parts at once.

Where do the matrices' numbers come from? Training — backpropagation pushes them toward whatever makes translation work, and we won't open that box on this page (the recipe lives in §5 of the paper). What we can do is run the mechanism itself, by hand, right now.

“its” has a query; every word has a key and a value. Time to run the lookup — with numbers small enough to check on paper.

We shrink the world to four words of our sentence — Law, perfect, its, application — and make the vectors 4-dimensional (the real model uses 64 per head; nothing about the procedure changes). The word asking is “its”, with query q = [2, 0, 2, 0]. The keys are: k(Law) = [2, 0, 2, 0], k(perfect) = [0, 2, 0, 0], k(its) = [1, 0, 1, 0], k(application) = [2, 0, 1, 0]. Notice we've rigged Law's key to point the same way as the query — that's the geometric meaning of “this key matches”.

Read the result. The new vector for “its” is about two-thirds Law-stuff and a quarter application-stuff — the pronoun has pulled in the words it refers to. Those are the two heavy arcs of Step 4, now with arithmetic under them. 0.66 and 0.24 aren't decoration; you just computed where they come from.

This one line is steps a–d in matrix clothes. Stack every word's query into a matrix Q, every key into K, every value into V: then QK^T computes all the dot products at once, the softmax runs per row, and the final multiplication by V does every word's blend simultaneously. No loop over positions anywhere — the Step 2 promise of parallelism is cashed in right here, as a pair of matrix multiplications.

In step b you divided by 2 on faith. The reason is a one-line statistics argument, and it's the paper's own (footnote 4).

Suppose each component of q and k is a random number with mean 0 and variance 1. Their dot product adds up d_k such products, so its variance is d_k — typical size √d_k. At our toy d_k = 4 that's a tame 2. At 512 it's about 23, and softmax of scores around ±23 is winner-take-all: one weight lands at ≈ 1.0, the rest at ≈ 0.0000000001. That's fatal for learning — in that regime the gradients through softmax are vanishingly small, so training stalls. Dividing by √d_k puts the variance back to 1, whatever the dimension.

You computed one blend for “its”: mostly Law, some application. Good — but a sentence holds more than one kind of relationship, and one blend can only encode one.

“its” doesn't just need its antecedent. Some layer needs to know it sits next to “application”; another that it's inside the clause after “but”. A single attention head must cram all of that into one set of weights — one weighted average — and the paper is blunt about the cost: averaging inhibits it (§2). The fix is to run several small attentions side by side. Project the words into h = 8 different 64-dimensional subspaces — each head gets its own W_i^Q, W_i^K, W_i^V — run equation (1) independently in each, then concatenate the eight outputs and mix them with one last matrix W^O.

Does the freedom get used? The paper's Figure 5 answers with our own sentence. Two heads, same layer, same input — completely different habits:

Multi-head attention was the hard half. One more part and you can read the paper's architecture diagram without a legend.

After attention has mixed information between positions, each position gets private processing: a two-layer feed-forward network, applied to every word separately and identically — FFN(x) = max(0, xW₁ + b₁)W₂ + b₂ — expanding from 512 up to d_ff = 2048 and back. Same weights at every position, different weights in every layer.

Around both sublayers runs the plumbing that makes a 6-deep stack trainable: a residual connection adds each sublayer's input to its output, and layer normalization re-scales the result — LayerNorm(x + Sublayer(x)). We'll treat LayerNorm as a black box that keeps the numbers well-behaved (it has its own paper, cited as [1]); the residual idea is borrowed from ResNet, and it's why d_model must stay 512 everywhere — you can only add vectors of the same width.

Track “its” through this drawing once. It arrives as a vector, leaves the attention block carrying its 0.66-Law blend, gets added to its old self (residual), normalized, then privately transformed by the FFN, added and normalized again — and hands the result to the next layer, where the whole game repeats with sharper inputs. After six rounds, “its” is no longer a pronoun; it's a pronoun that knows its noun.

One thing in the tower should bother you: nothing so far ever used word order. Shuffle our sentence and every dot product in Step 5 comes out identical.

Recurrence got order for free — it read in order. With recurrence gone, order has to be smuggled into the data itself. The paper adds a positional encoding vector to each word's embedding before the first layer: position pos, dimension i gets sin(pos / 10000^2i/d_model) in the even slots and the matching cosine in the odd ones. Each dimension is a wave; the wavelengths grow geometrically from 2π to 10000·2π. Fast waves distinguish neighbors, slow waves distinguish regions — together, every position gets a unique fingerprint:

Why sines instead of just learning a vector per position? The authors give two reasons, one proved and one hoped. The proved one: for any fixed offset k, PE_pos+k is a linear function of PE_pos, which should make “attend three words back” an easy pattern to learn. The hoped one: waves extend to any position, so the model might handle sentences longer than any it saw in training. Honest footnote — they tried learned position embeddings too, and the scores came out nearly identical (Table 3, row E). The sinusoids won on the extrapolation hope, not on the scoreboard.

The encoder is done: 21 English vectors, each soaked in context. Now the other tower has to write German — and writing is where attention needs a leash.

Fair warning: this is the twistiest part of the architecture, because the decoder's input is its own earlier output. During training, the model learns to predict every position of the real German sentence at once — position i should be predictable from positions before i only. But self-attention, as built in Step 5, looks everywhere — including at the very future words the model is supposed to predict. Unleashed, it would learn to copy the answer sheet.

The fix costs one matrix: before the softmax, set every query-to-future-key score to −∞. Softmax turns −∞ into weight 0 — the future is still on the page, but it gets exactly none of the blend. Combined with shifting the output sequence one slot right (so position i's input is word i−1), the prediction for position i can only draw on what comes before it.

The full decoder layer stacks three attentions, and you've now met every one of them. The paper's §3.2.3 lists the trio: encoder self-attention (Steps 4–7, unrestricted), decoder masked self-attention (the grid above), and between them encoder–decoder attention — queries come from the German side, keys and values from the encoder's 21 English vectors. That third one is where translation actually crosses the language line: when the decoder writes “wird”, its query reaches into the English sentence and lands on the key of “will”. Same equation (1) every time; only the sources of Q, K, V change.

The machine is complete: encoder, decoder, masks, waves. The claim in the title still needs numbers behind it.

Translation quality is scored in BLEU — an overlap score between the system's output and human reference translations, where higher is better and a couple of points is a big deal. On WMT 2014 English→German, the big Transformer scores 28.4: more than 2 BLEU above every previous system, including ensembles of models. Even the base model beats every prior single model and ensemble — and that's the cheap one:

The parsing line deserves a pause. The authors pointed the same architecture at a different, structurally strict task — producing parse trees — changed almost nothing, and still landed near the state of the art (91.3 F1 training on just 40K sentences, 92.7 semi-supervised). For a paper claiming attention is all you need, generalizing beyond translation was the claim's first real test outside its home turf.

Results say the whole machine works. Ablations say which parts were load-bearing: the authors re-trained the base model dozens of ways and watched what each change cost.

The scores below are on the development set (newstest2013), so they're a notch lower than Step 11's test numbers — what matters is the gaps:

Reading the rows: one head loses 0.9 BLEU against the best setting — Step 7's specialization argument, measured — but 32 heads drops quality too, so more isn't monotonically better. Shrinking the keys to d_k = 16 hurts, which the authors read as a hint that judging query–key compatibility is genuinely hard — a dot product may not even be the final word. Dropping dropout costs 1.2 BLEU, the single worst change. And learned position embeddings tie the sinusoids almost exactly, which is the honest footnote under Step 9.

Now close the loop. Five questions, answers folded away — each one traces back to a step you did with your own hands: