Skip to content
POLARISAI ACADEMY
R08-101 · Navigator

Running Models Locally: Hardware, Quantization and Engines

You can add weight bytes, KV-cache bytes, scratch bytes, and a fixed runtime reserve for a stated model, round a weight onto an INT4 grid, and check that a local server returns the same text twice when temperature is 0.

Level
Navigator
Time
22 h (estimate)
Prerequisites
F01–F07, R02-101
Course 1 of 4
Jump to

After this course you can

The student can calculate exact VRAM and KV-cache memory requirements for any model architecture and quantization level, and configure an offline local inference server that passes API contract and determinism tests.

The license file beside the weights

Open-weight means the weight file can be downloaded and run on your own machine. The license file in that same folder is the rule for the copy you hold. Two widely used texts are the Apache-2.0 license and the MIT license. A community license is a further text written by the publisher of that particular file, and it can add a cap the other two leave out. You follow the file that shipped with the weights. A label on a download page becomes a rule only after you open that text.

Apache-2.0 gives a copyright license to use, copy, modify, and distribute the weights, including inside a product you sell. The text also includes a patent license from each contributor covering that contributor's contribution. When you pass a copy on, you include the Apache-2.0 license, you keep the attribution lines from the NOTICE file, and you mark files you changed. The license leaves the publisher's trademark with the publisher. A worked folder has three files: weights.gguf, LICENSE with the Apache-2.0 text, and NOTICE with the line Copyright 2024 North Workshop. Copying that folder to a second machine is allowed when LICENSE and NOTICE travel with the weights. MIT gives permission to use, copy, modify, merge, publish, distribute, sublicense, and sell, and it asks you to keep the copyright line and the permission notice with every copy or substantial portion. The MIT text has no patent clause. Community-Local is a license defined only in this lesson so the cap is a number you can check. Local running, modification, and giving copies away are allowed when the license file stays with the copy. A hosted service is allowed up to 100 distinct users in a month. User 101 in that same month requires a separate written agreement. Apache-2.0 and MIT, as those texts are written, set no user-count cap.

Four terms that fill VRAM

VRAM is the memory on the card. It has to hold the weights and the working store for the reply. In this lesson 1 MiB means 2^20 bytes, which is 1,048,576 bytes, and 1 GiB means 1024 MiB. Model A has N = 2^31 parameters, written out as 2,147,483,648. It has L = 24 layers, n_kv = 8 key-value heads, d_head = 64, and d_model = 2048. The worked sequence length is S = 1024 tokens, and the batch is B = 1. Weight bytes equal N times the bytes stored for one weight: 2 for FP16, 2 for BF16, 1 for INT8, and 1/2 for plain INT4 codes. FP16 and BF16 both give 2,147,483,648 times 2 = 4,294,967,296 bytes, which is 4 GiB, or 4096 MiB. The byte count matches. The layouts differ: FP16 uses 5 exponent bits and 10 fraction bits, plus a sign bit, while BF16 uses 8 exponent bits and 7 fraction bits, plus a sign bit.

The KV cache stores one key vector and one value vector for every layer and every token still in the sequence. Its size is 2 times L times n_kv times d_head times bytes_act times S. With bytes_act = 2 that product is 2 times 24 times 8 times 64 times 2 times 1024 = 50,331,648 bytes = 48 MiB. Scratch bytes are the two live activation buffers in this lesson's accounting: B times S times d_model times bytes_act times 2 = 1 times 1024 times 2048 times 2 times 2 = 8,388,608 bytes = 8 MiB. The runtime reserve is a fixed 512 MiB for the engine's own context. Add the four terms for FP16: 4096 + 48 + 8 + 512 = 4664 MiB. Plain INT8 weights, with the same FP16 cache and scratch, give 2048 + 48 + 8 + 512 = 2616 MiB. Plain INT4 weights give 1024 + 48 + 8 + 512 = 1592 MiB. Raise S from 1024 to 2048 and both the cache and the scratch double, to 96 MiB and 16 MiB, because each term is linear in S. The weight term and the 512 MiB reserve stay as they are. Plain INT4 at S = 2048 is 1024 + 96 + 16 + 512 = 1648 MiB.

Quantization grids and three packings

Quantization stores each weight as a code on a coarse grid and multiplies that code by a scale when the model runs. On this lesson's symmetric INT4 grid the codes run from -7 to 7 and the scale is s = 0.05. Divide the weight by s, take the nearest integer, and clip anything past 7 or below -7 to the end of the grid. When the fractional part is 0.5 or larger, round to the integer farther from zero. The four weights 0.20, 0.27, 0.13, and 0.05 land like this. 0.20/0.05 = 4 exactly, stored as 0.20, absolute error 0. 0.27/0.05 = 5.4, which rounds to 5 and stores 0.25, absolute error 0.02. 0.13/0.05 = 2.6, which rounds to 3 and stores 0.15, absolute error 0.02. 0.05/0.05 = 1 exactly, stored as 0.05, absolute error 0. The mean absolute error is 0.04/4 = 0.01. A weight of 0.50 gives 0.50/0.05 = 10, clipped to 7, stored as 0.35, absolute error 0.15. A weight of 0.33 gives 0.33/0.05 = 6.6, which rounds to 7 and stores 0.35, absolute error 0.02. On a four-position check the probability of the true next token is 1/2, 1/2, 1/4, and 1/4. The bit cost is the average of the base-2 negative logs: 1, 1, 2, and 2, so the mean is 1.5 bits. After the coarser grid those probabilities are 1/2, 1/4, 1/4, and 1/8, the costs are 1, 2, 2, and 3, and the mean is 2.0 bits. The rise on this file is 0.5 bits per token. The probabilities are picked so every log is a whole number.

GGUF is a file that stores the quantized tensors and a header with L, n_kv, d_head, d_model, and N. A Q4_0 block holds 32 four-bit codes and one 2-byte scale. The scale adds 2/32 = 1/16 byte per weight. For Model A the extra is 2,147,483,648/32 times 2 = 134,217,728 bytes = 128 MiB, so the Q4_0 weight term is 1024 + 128 = 1152 MiB. The four-term sum with that weight term, at S = 1024, is 1152 + 48 + 8 + 512 = 1720 MiB. AWQ, activation-aware weight quantization, still spends about 4 bits on each weight, and it aims the grid at channels whose activations are large. Take w = 0.13 and an activation of 10. The true product is 1.30. The plain grid stores 0.15, the product is 1.50, and the error is 0.20. Multiply the weight by alpha = 2 before the grid: 0.26/0.05 = 5.2, which rounds to 5 and stores 0.25. Divide by alpha afterward and the weight is 0.125. The product is 1.25, and the error falls to 0.05. A packing that shares one 2-byte scale across 128 weights adds 2,147,483,648/128 times 2 = 33,554,432 bytes = 32 MiB, so that weight term is 1024 + 32 = 1056 MiB. EXL2 picks a bit width per layer so the average hits a target. Split Model A into two halves of 2^30 parameters each. One half at 8 bits and one half at 4 bits uses 2^30 times 1 byte plus 2^30 times 1/2 byte = 1.5 GiB = 1536 MiB of codes, and the average is 6 bits per weight.

Bandwidth, watts, and the token ceiling

After the four-term sum fits on the card, a single stream is limited by how fast the card can read the weights. This lesson's decode ceiling at batch 1 is delivered bandwidth divided by the weight size, treating the KV and scratch reads as small beside the weights. Card C delivers 400 GiB/s and has a 200 W power cap. FP16 weights are 4 GiB, so the ceiling is 400/4 = 100 tokens per second. Plain INT4 weights are 1 GiB, so the ceiling is 400/1 = 400 tokens per second. The lesson's power assumption is that delivered bandwidth scales with the power ratio when the cap is lowered. A cap of 150 W is 150/200 = 0.75 of the rated power, so the bandwidth becomes 300 GiB/s. The FP16 ceiling falls to 300/4 = 75 tokens per second. The plain INT4 ceiling falls to 300/1 = 300 tokens per second. One hour at 200 W uses 0.2 kWh. One hour at 150 W uses 0.15 kWh. The cooler has to move those watts into the room. When the cooler falls behind, the card settles on the lower cap and the token ceiling falls with the bandwidth.

A batching server reads the weights once per step for the whole batch. At batch 4 the FP16 weight read that supported 100 tokens per second on one stream supports 400 tokens per second across the four streams together, at the 400 GiB/s figure. Each stream keeps its own KV cache, so the cache term becomes 4 times 48 MiB = 192 MiB at S = 1024, while the weights stay one copy. Scratch also follows B, so it becomes 4 times 8 MiB = 32 MiB. The batch-4 FP16 sum is 4096 + 192 + 32 + 512 = 4832 MiB. Card C in the worked case has 8192 MiB of VRAM. The batch-1 plain INT4 sum of 1592 MiB leaves 8192 - 1592 = 6600 MiB free. The batch-1 FP16 sum of 4664 MiB leaves 8192 - 4664 = 3528 MiB free.

Three local engines and one stability check

Once the files are on disk, Model A can be served with no further download. Ollama, a one-command runner, keeps the weight file in its own store and listens on 127.0.0.1 port 11434. The GGUF command-line engine takes the path to a .gguf file and the context length, and it can open a port on the same machine. vLLM is a batching server: several requests share one read of the weights, which is the batch-4 case above, and its usual port is 8000. Any of the three can answer a local request that carries a prompt, a temperature, and a maximum of 16 new tokens. With the weights already local, a container started with network access turned off still answers, because the reply is computed from the file on disk.

The stability check sends the same prompt twice. At temperature 0 every step keeps the single highest token, so the two reply strings match on the same file and the same build. At temperature 0.7 the seed selects the draw: two calls with seed 7 match each other, and a call with seed 8 can leave that text. You compare the strings yourself. A match at temperature 0 supports the claim that this build, this file, and this prompt are stable. A mismatch at temperature 0 means the engine is still sampling, or that the file or the build changed between the two calls.

An example

Model A on Card C, plain INT4, by hand

Card C has 8192 MiB of VRAM, delivers 400 GiB/s, and is capped at 200 W. Load Model A as plain INT4 weights with FP16 KV cache and FP16 scratch, S = 1024, B = 1. Weight bytes: 2,147,483,648 times 1/2 = 1,073,741,824 bytes = 1024 MiB. KV cache: 50,331,648 bytes = 48 MiB. Scratch: 8,388,608 bytes = 8 MiB. Runtime reserve: 512 MiB. The sum is 1024 + 48 + 8 + 512 = 1592 MiB. Free memory is 8192 - 1592 = 6600 MiB, so the file fits. The batch-1 ceiling is 400 GiB/s divided by 1 GiB = 400 tokens per second.

The same card with FP16 weights uses 4096 + 48 + 8 + 512 = 4664 MiB and leaves 8192 - 4664 = 3528 MiB free. The ceiling drops to 400/4 = 100 tokens per second because every new token reads 4 GiB. The folder next to those weights holds an Apache-2.0 LICENSE and a NOTICE line, Copyright 2024 North Workshop. A copy on a second machine keeps both files, and a changed quant config is marked in the file you changed. Community-Local is a different text: at 100 users in a month the hosted service is inside the cap, and at 150 users it requires a separate written agreement.

Exercise

Check it on paper

Using Model A and this lesson's four terms, write the plain INT8 sum in MiB at S = 1024 and B = 1. Write the plain INT4 sum at S = 2048. Write the Q4_0 weight term and the Q4_0 four-term sum at S = 1024. Round the weight 0.33 on the grid s = 0.05 and give the stored value and the absolute error. Give the FP16 batch-1 ceiling in tokens per second when Card C is cut to 300 GiB/s, and the FP16 aggregate ceiling at batch 4 when the card still delivers 400 GiB/s. Say what Community-Local requires at 150 distinct users in one month. Give the mean bit cost, in bits, of the coarser four probabilities 1/2, 1/4, 1/4, and 1/8.

The plain INT8 sum is 2048 + 48 + 8 + 512 = 2616 MiB. The plain INT4 sum at S = 2048 is 1024 + 96 + 16 + 512 = 1648 MiB. The Q4_0 weight term is 1152 MiB, and the four-term sum is 1152 + 48 + 8 + 512 = 1720 MiB. The weight 0.33 becomes 0.33/0.05 = 6.6, which rounds to 7 and stores 0.35, with absolute error 0.02. The FP16 ceiling at 300 GiB/s is 75 tokens per second, and the batch-4 aggregate ceiling at 400 GiB/s is 400 tokens per second. Community-Local requires a separate written agreement at 150 users, because the cap is 100. The coarser mean bit cost is 2.0 bits.

Check questions

Answer on your own first. Then open a question to see the answer.

What is the four-term plain INT4 sum for Model A at S = 1024 and B = 1, in MiB, and which four numbers are added?

1592 MiB, from 1024 MiB of weights, 48 MiB of KV cache, 8 MiB of scratch, and 512 MiB of runtime reserve.

Why do the FP16 weights and the BF16 weights of Model A both occupy 4096 MiB, and how many exponent bits does each layout use?

Both store 2 bytes per parameter, so both are 4 GiB. FP16 uses 5 exponent bits. BF16 uses 8 exponent bits.

How many MiB do the Q4_0 scales add for Model A, and what is the four-term sum at S = 1024 when the weight term includes those scales?

The scales add 128 MiB. The weight term is 1152 MiB, and the four-term sum is 1720 MiB.

Why do two calls at temperature 0 return the same text, and what is the batch-1 FP16 token ceiling on Card C at 400 GiB/s?

Each step keeps the single highest token, so the strings match on the same file and build. The ceiling is 100 tokens per second.

Take with you

The model fits when the four-term sum is inside the card's MiB, one stream's speed is the delivered GiB/s divided by the weight GiB, and the license file beside the weights is the rule for a second copy or a hosted service.

Final task

Pass a 25-item keyed VRAM, bandwidth, and quantization calculation benchmark, and submit a local inference configuration and benchmark script that passes 20 automated tests in a zero-network (--network=none) container with a tiny fixture model.

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist