The first surprise when people run a model locally is that “it fits in memory” and “it runs well” are different questions. Weight size gives a rough idea of whether the model can fit. Responsiveness depends on more: context, KV cache, concurrency, runtime overhead and the hardware's ability to move data.
For a quick weights-only estimate, multiply parameter count by bits per parameter and divide by eight. An 8-billion-parameter model at 16 bits is about 16 billion bytes, or 14.9 GiB. At 4 bits, the raw figure is about 4 billion bytes, or 3.7 GiB. These numbers are a starting estimate, not the full memory requirement.
Actual memory use also includes quantization metadata, runtime buffers, intermediate values and context-related state. Training adds further requirements such as gradients and optimizer state. On a workstation, leave enough memory for the operating system and other applications as well.
Transformer inference usually has two noticeably different phases. Prefill processes the input context. Decode generates output tokens one by one. Prefill often leans more heavily on compute, while decode can become limited by memory movement at low batch sizes. The real bottleneck depends on the model, hardware and workload, so profile it rather than assuming. [11]
Longer active sequences and more concurrent requests put additional pressure on the KV cache, which stores attention keys and values from earlier tokens. Cache strategies trade memory, precision and speed. Quantizing model weights reduces one part of the memory budget, not all of it. [10]
Batching can improve throughput by sharing hardware work across requests, but queueing and large batches may increase an individual's wait. Splitting a model across accelerators adds communication, which makes interconnect performance relevant. Offloading to slower memory may allow a model to fit while making it slower.
Measure the workload your users will actually create: time to first token, generation rate, end-to-end latency, peak memory and throughput at realistic concurrency. Record input and output lengths. A benchmark built around one short prompt says little about a service dominated by long documents.
At facility scale, the same principle continues. Power, cooling, rack capacity and networking constrain what can be sustained. CUDA and other accelerator software stacks matter when performance becomes your responsibility. For most application teams, the practical question is simpler: can the required quality and latency be delivered within a budget the team can operate?
These five bounded exercises turn the concepts in the book into evidence you can inspect. Use public or synthetic data and an approved environment. The goal is not to finish a lab quickly, but to understand why the result succeeded or failed.
- 27 Practise with evidence
- 28 Practise actions and failure recovery