Series 01 / Pre-training models
Build the base model
Fifteen focused chapters move from text representation and data design to architecture, sparse experts, optimization, distributed training, and reproducible evidence.
- 01Read ↗
What a Language Model Learns
Token representations, autoregressive factorization, teacher forcing, cross-entropy, and the limits of next-token prediction.
- 02Read ↗
From Unicode to Byte-Pair Encoding
Code points, UTF-8 bytes, merge learning, vocabulary construction, compression, and exact encode–decode behavior.
- 03Read ↗
Corpus Design and Data Lineage
Source registration, parsing, filtering, deduplication, decontamination, mixture weights, document boundaries, and immutable shards.
- 04Read ↗
Batches, Tensor Shapes, and Shifted Targets
Turn token streams into input–target windows, track batch and sequence dimensions, mask invalid positions, and prevent split leakage.
- 05Read ↗
The Transformer Residual Stream
Follow embeddings through encoder, decoder, and encoder–decoder families while keeping every tensor contract explicit.
- 06Read ↗
Queries, Keys, Values, and Causal Masks
Build scaled dot-product attention, understand multi-head projections, apply causal and padding masks, and verify the softmax axis.
- 07Read ↗
Residual Blocks, Normalization, and MLPs
Assemble pre-norm attention and feed-forward branches, preserve gradient paths, and reason about width, depth, and activation choice.
- 08Read ↗
Position and Context Length
Compare learned positions, sinusoidal encodings, rotary embeddings, relative bias, extrapolation, and the real cost of longer context.
- 09Read ↗
KV Caches and Efficient Attention
Separate full-sequence training from incremental decoding, account for cache memory, and test fused kernels against a clear reference.
- 10Read ↗
Scaling Laws and Training Budgets
Allocate parameters, clean tokens, context, batch size, compute, and wall-clock budget without confusing size with capability.
- 11Read ↗
Mixture-of-Experts Routing
Map tokens to experts with top-k routing, distinguish total from active parameters, and trace dispatch and combine operations.
- 12Read ↗
Expert Capacity, Balance, and Parallelism
Handle overflow, token dropping, auxiliary losses, shared experts, expert parallel communication, and routing failure modes.
- 13Read ↗
Initialization and Optimization
Set residual-aware initialization, AdamW parameter groups, warmup and decay schedules, gradient accumulation, and clipping.
- 14Read ↗
Distributed Training and Throughput
Use mixed precision, fused kernels, compilation, data parallelism, all-reduce, and exact global-token accounting.
- 15Read ↗
Checkpoints, Evaluation, and Reproducibility
Package weights with optimizer and data state, compare fixed-token runs, audit contamination, and preserve the evidence behind every result.
Series 02 / Post-training models
Shape the model with feedback
Demonstrations, preferences, rewards, graders, environments, and trajectories that turn a base model into a capable assistant.
- 01Read ↗
Post-training a Language Model
Assistant contracts, conversation data, preferences, verifiable RL, PPO, DPO, tools, working memory, security boundaries, and evaluation.
- 02Read ↗
Evals and RL Environments Are the Same Machine With Different Jobs
Prompt, environment, grader: the architecture is shared. The optimization loop is not.
- 03Read ↗
Useful Tasks Live at the Edge of Capability
Too easy produces no signal. Too hard produces only zeros. Progress lives in the narrow band between them.
- 04Read ↗
A Grader Is Never Just Measuring the Model
Every scoring rule creates pressure. The model learns the proxy—especially when the proxy is easier than the goal.
- 05Read ↗
Real Users Have Higher Entropy Than Simulated Ones
Why model-generated users repeat familiar paths—and miss the strange actions that expose real failures.
- 06Read ↗
Long-Horizon Agents Need More Than More Context
A larger window delays forgetting. It does not create judgment, durable memory, or a model of the user.
- 07Read ↗
Continual Learning Is the Missing Layer
Files full of notes are not experience. What would it take for an agent to genuinely get better at its workplace?
- 08Read ↗
Taste Is Becoming the Bottleneck in Model Evaluation
The easy capabilities have abundant data. What remains is difficult precisely because correctness requires judgment.
- 09Read ↗
AI Will Change Software Engineering Before It Replaces Engineers
Automation moves the bottleneck upward—from writing syntax to designing systems, modeling users, and deciding what matters.
- 10Read ↗
What a Good Agent Trajectory Must Capture
The state, actions, evidence, provenance, and costs required to explain why an outcome happened.
Series 03 / Model serving
Put the model into production
Ten focused chapters turn a checkpoint into a reliable service by connecting latency objectives, request execution, cache economics, scheduling, parallelism, numerical formats, and operational evidence.
- 01Read ↗
Inference Begins With a Service Contract
Define time to first token, inter-token latency, end-to-end latency, throughput, availability, workload shape, and cost before selecting hardware.
- 02Read ↗
Anatomy of an Inference Request
Trace admission, tokenization, queueing, batching, model execution, sampling, streaming, cancellation, and observability across explicit boundaries.
- 03Read ↗
Prefill and Decode Are Different Workloads
Separate compute-heavy prompt processing from memory-bound token generation and measure each phase with the metric it controls.
- 04Read ↗
KV-Cache Arithmetic and Memory
Calculate cache bytes per token and request, then connect context length, concurrency, paging, eviction, and admission control.
- 05Read ↗
Continuous Batching and Scheduling
Schedule token work rather than whole requests, control queue delay, handle preemption and cancellation, and prevent long prompts from dominating.
- 06Read ↗
Parallel Inference Across Accelerators
Choose tensor, pipeline, data, or expert parallelism by balancing memory fit, communication volume, latency, throughput, and failure domains.
- 07Read ↗
Quantization and Numerical Formats
Compare weight, activation, and cache precision; measure memory traffic and kernel support while guarding output quality and stability.
- 08Read ↗
Speculative Decoding and Latency Levers
Use fused kernels, prefix reuse, speculative decoding, and workload-aware tuning only after identifying the phase that dominates latency.
- 09Read ↗
Capacity Planning and Reliable Overload
Turn memory, token throughput, queue limits, timeouts, retries, backpressure, and degraded modes into an explicit capacity envelope.
- 10Read ↗
Load Testing and the Evidence Bundle
Benchmark realistic prompt and output distributions, concurrency, streaming, cache pressure, failures, tail latency, quality, and cost together.
About
Researcher’s curiosity.
Engineer’s discipline.
I’m Vikram Kharvi, an applied AI researcher and engineer in California. I work across the model lifecycle—from pre-training systems and post-training environments to evaluation, deployment, and monitoring.
These field notes organize what I am learning while building reliable foundation-model systems, with a particular focus on reinforcement-learning environments and production inference.
I hold a Master of Artificial Intelligence from Northwestern University and a Bachelor of Information Science Engineering from PES University.