Large language models: the system map.

The model sees token IDs, not text.

The tokenizer maps a byte sequence to a variable-length sequence of integers. An embedding table turns each integer into a vector of width dmodel. This representation controls sequence length, vocabulary size, memory use, and how much raw text fits inside a fixed token budget.

Tokenization is therefore part of the architecture. A model with a poor representation spends context and compute reconstructing fragments that a better vocabulary would express compactly.

The training mask determines what kind of language model you build.

Encoder-only models use bidirectional context to produce contextual representations for classification, tagging, and span prediction. Decoder-only models use a causal mask and learn next-token prediction, making them the dominant foundation for open-ended generation. Encoder–decoder models encode a source sequence bidirectionally and generate a target sequence causally, which is useful when the task naturally maps one sequence to another.

“Large language model” does not imply one family, one modality, or one deployment shape. Parameter sharing, attention variants, sparse experts, context length, and training mixture create a broad design space. The correct family follows from the learning contract and the behavior the model must support.

One objective produces millions of local prediction tasks.

For a token sequence, an autoregressive model predicts each token from the prefix before it. Teacher forcing evaluates every valid position in parallel during training. The output head maps hidden states to vocabulary logits; cross-entropy rewards probability assigned to the observed next token.

The objective is simple but the learned behavior is data-shaped. Source mixture, duplication, contamination, language balance, and document boundaries decide which conditional patterns the model sees repeatedly. Masked objectives, span corruption, contrastive objectives, and multimodal prediction change what information the representation must preserve, even when the underlying Transformer family is similar.

Parameters, tokens, and compute form one budget.

Increasing parameter count raises representational capacity. Increasing clean training tokens supplies more evidence. Increasing compute buys more optimizer steps or a larger network. Scaling one axis while starving another produces an undertrained or overfit model.

Scaling laws are empirical planning tools, not magic exponents. Fit them on smaller runs, hold architecture and data policy stable, and estimate which allocation of parameters and tokens minimizes loss under a fixed compute budget. Extrapolation fails when data quality, optimizer behavior, context length, or hardware utilization changes outside the measured regime.

A scaling run should report exact tokens processed, non-padding fraction, achieved throughput, optimizer steps, parameter count, and wall-clock cost. Parameter count alone cannot explain the training budget. Dense parameter count, activated parameter count, training tokens, and effective data quality should be reported separately when comparing architectures.

A checkpoint is more than a weight file.

To continue a run exactly, save model weights, optimizer moments, learning-rate state, random generators, data cursor, global token count, tokenizer version, and configuration. To reproduce behavior, also save numerical mode, code revision, dependency versions, and generation defaults.

The checkpoint selected for release need not be the final step. Held-out loss, domain slices, downstream tasks, stability, and operational measurements should determine which artifact is kept. Quantization, distillation, adapters, and post-training create derivative artifacts; their lineage should point back to the exact base checkpoint and tokenizer.

A base model needs an evidence bundle.

Track held-out loss by domain and language, memorization probes, calibration, few-shot task behavior, generation diversity, training throughput, memory pressure, and run failures. Compare checkpoints at the same token budget and sampling policy. Otherwise, a decoding change can be mistaken for a weight improvement.

Capability is jagged rather than uniformly ordered. A model may solve a hard coding task yet fail a simple counting, spelling, or instruction-placement test. Aggregate scores hide that shape. Keep behavioral slices, adversarial variants, and concrete failure examples beside every headline number.

Evaluation should separate the base model from the decoding policy and any later behavioral tuning. The same checkpoint can look very different under different prompts, temperatures, stop rules, and context construction.