Transformers: the decoder block.

One block pattern supports three major architectures.

An encoder stack uses bidirectional self-attention and returns one contextual state per input position. A decoder stack uses causal self-attention and predicts the next token from the prefix. An encoder–decoder model combines both: the decoder adds cross-attention whose queries come from the target stream while keys and values come from encoder states.

Vision Transformers replace text tokens with image patches; multimodal Transformers add modality-specific encoders or project several modalities into one token stream. The attention operation is reusable, but masks, position schemes, and output objectives change the contract.

Track one tensor shape through the network.

Token embeddings enter the stack as X ∈ ℝᴮˣᵀˣᵈ. The batch size B and sequence length T stay fixed across a decoder block; the model width d is restored after both attention and the MLP. This stable outer shape makes blocks stackable.

Shape assertions belong next to the operations. Most attention bugs are not conceptual; they are silent transposes, broadcasts, or masks applied along the wrong axis.

Start with averaging so every matrix operation has a meaning.

The from-scratch lecture develops attention by replacing a bigram model with progressively stronger context aggregation. The weakest version averages all previous token vectors. A Python loop makes the rule obvious; a lower-triangular matrix multiplication performs the same aggregation in parallel. Normalizing each row turns the triangle into averaging weights.

Replacing uniform weights with learned affinities creates content-dependent aggregation. A token emits a query describing what it needs and a key describing what it offers. Their dot product becomes an affinity. Values carry the information actually aggregated.

# pedagogical path from averaging to learned attention
wei = tril(ones(T, T))
wei = wei / wei.sum(dim=1, keepdim=True)
uniform_context = wei @ x

q, k, v = Wq(x), Wk(x), Wv(x)
learned_context = softmax(q @ k.T + causal_bias) @ v

This construction exposes four contracts that compact library calls can hide: tokens communicate only within the same batch element; causal masking determines which positions are legal; attention by itself does not know token order; and the value path is separate from the query–key addressing path.

Queries address; keys match; values carry content.

Learned projections turn each token representation into query, key, and value vectors. The scaled dot product compares every query with eligible keys. Softmax converts scores into row-wise weights, and the weighted values become the head output.

Multi-head attention repeats this calculation across h smaller subspaces, concatenates the results, and projects them back to width d. Heads do not come with predefined roles; any specialization is learned through the training objective. A head can attend broadly, locally, or to a syntactic relation, but those patterns are outcomes of optimization rather than hard-coded modules.

q, k, v = project(x)                       # [B, h, T, dh]
scores = q @ k.transpose(-2, -1) / sqrt(dh)
scores = scores.masked_fill(causal == 0, -inf)
weights = softmax(scores, dim=-1)
out = merge_heads(weights @ v) @ W_o        # [B, T, d]

Causality is enforced before softmax.

At target position t, the model may use positions ≤ t but never positions to the right. A lower-triangular mask assigns a large negative bias to prohibited scores so their softmax probability becomes zero. The training loss then compares position t with token t+1.

The factor 1/√dh controls the variance of query–key dot products. Without it, wider heads produce larger logits, softmax rows become extremely sharp, and gradients through most positions shrink. Scaling keeps the initial attention distribution in a trainable regime.

Padding masks solve a different problem: they prevent synthetic padding positions from contributing to attention or loss. Document packing may require an additional block mask so one document cannot see tokens from the next packed document.

Residual pathways make depth trainable.

In a pre-normalized decoder, normalization occurs before each sublayer. The sublayer output is added to the residual stream. This gives gradients a direct path through the stack and keeps every block responsible for a learned update rather than a complete replacement.

GPT-2 uses a closely related residual design with layer normalization around its attention and MLP paths. Modern implementations often prefer pre-normalization because the identity stream remains especially clear through deep stacks. Whichever ordering is chosen, match the checkpoint exactly when loading weights; norm placement is part of the function, not a cosmetic refactor.

The MLP expands each position to a larger hidden width, applies a nonlinearity or gate, then projects back to d. It processes positions independently; attention is the component that communicates across positions. In many models the MLP owns more parameters than attention, which is why it is also the usual location for sparse experts. Gated activations multiply a content branch by a learned gate before projecting back, giving the block a richer position-wise transformation.

Attention needs an explicit notion of order.

Without positional information, attention is equivariant to token permutation. Absolute embeddings add a position vector to each token. Relative schemes modify scores using token distance. Rotary position embeddings rotate query and key coordinates as a function of position so their dot product carries relative displacement.

Context extension changes more than a configuration number. The position scheme, attention memory, training examples, and evaluation lengths must support the new range.

Generation reuses keys and values from the prefix.

During autoregressive generation, earlier token representations do not change because the causal mask prevents them from seeing later tokens. Each layer can store the projected keys and values from the prefix. The next step computes only one new query, key, and value, appends the new key and value, and attends over the accumulated cache.

The cache converts repeated projection work into persistent memory. Its size grows with batch, layers, sequence length, KV heads, and head width. Grouped-query or multi-query attention reduces cache size by sharing key/value heads across multiple query heads.

Attention design is a memory-and-movement decision.

Full self-attention forms interactions between every pair of sequence positions, so the score matrix grows quadratically with sequence length. The mathematical operation may stay the same while a fused, tiled kernel changes how intermediate values move through memory. FlashAttention follows this principle: compute exact attention in blocks, retain only the statistics required for a stable softmax, and avoid materializing the full score matrix in high-bandwidth memory.

Long context is therefore a complete contract: position encoding must remain stable, the training mixture must contain useful long examples, kernels must fit the memory budget, and evaluations must test whether information is actually used across distance. A configured maximum length is not evidence of effective long-context behavior.