AI Builder
AI Builder turns three choices — model size, how tightly each weight is stored, and how long a conversation you keep — into a memory estimate and a hardware tier. The weight estimate is the parameter count times the bytes stored for each weight, times about 1.2, plus a context cache that grows with the conversation.
Beginner
Use the three controls
The builder takes three inputs: model size in parameters, quantization, and context length. A parameter is one number stored in the model. A name such as 8B means about 8 billion of those numbers. Quantization is how many bits you keep for each number. Fewer bits make a smaller file, usually with a small drop in quality. FP16 keeps a 16-bit floating-point value, 2 bytes per parameter. Q8 keeps about 1 byte. Q4 keeps about half a byte. Context length is how many tokens stay active. A token is a piece of text, often a short word or part of a word.
Set 8 billion parameters, Q4, and 8,000 tokens. Read the memory line, then the tier: a laptop with about 16 GB, a home PC with a 24–32 GB graphics card, DGX Spark with 128 GB of unified memory, or DGX Station. Unified memory means the processor and the graphics chip share one pool. NVIDIA lists 128 GB for DGX Spark and support for models up to 200 billion parameters on one unit.
Multiply parameters by bytes per weight, then by about 1.2, and add the context cache. The 1.2 is spare room while the program runs. The same 8B model in FP16 is about 16 GB of weights before that spare room, which already fills the laptop band. Shared build lists open later.
Check an 8B model on a 16 GB laptop
I have a laptop with 16 GB of memory. Using this rule — memory is about parameter count times bytes per weight times 1.2, plus extra memory for the conversation — tell me whether an 8 billion parameter model fits at Q4 (0.5 byte per weight) and at FP16 (2 bytes). Leave 4 GB for the operating system and assume a short chat of a few thousand tokens. Say which setting fits, and what I should try first in a local app such as Ollama or llama.cpp.
Intermediate
The memory sum, worked out
Parameter count times bytes per weight is the size of the stored numbers. EleutherAI's inference rule, which Hugging Face cites, multiplies that size by about 1.2 for overhead while the model runs. Hugging Face's estimator only measures the memory needed to load the model; running it needs more.
Add the context cache, also called the KV cache: keys and values kept for tokens already read, which avoids a second attention pass. The cache grows with context length, layer count, and key-value heads. Two models with the same parameter count can need different caches. A fit at 2,000 tokens can fail at 32,000.
These figures are decimal gigabytes of weights, before the cache. 8B at Q4 is 4 GB, about 4.8 GB after the 1.2, inside a 16 GB laptop if the cache stays modest. 8B at FP16 is 16 GB, about 19.2 GB after the 1.2, over the laptop band. 32B at Q4 is about 19.2 GB after the 1.2, so a 24 GB card holds the weights until a long context pushes it over. 70B at Q4 is about 42 GB after the 1.2, past a 32 GB card and inside Spark's 128 GB. 70B at FP16 is about 168 GB after the 1.2, past one Spark and inside the Station's 252 GB of HBM3e.
llama.cpp files such as Q4_K_M sit near 4 bits per weight and a little above parameter count times 0.5, because of extra tensors. A mixture-of-experts model runs only some weights per token, and the full file still has to load.
Place a 70B model on a memory tier
Estimate inference memory for a 70 billion parameter model at Q4 (0.5 byte per parameter) and at FP16 (2 bytes). Compute weights as parameters times bytes, then multiply by 1.2, and label the result as before the KV cache. Compare both totals with 32 GB, 128 GB, and 252 GB. Then list the model-config fields you still need (layers, key-value heads, head size, bytes per cache element) to estimate the KV cache at 8,000 tokens. Do not invent a cache size without those fields.
Expert
Station memory, speed, power, and license
NVIDIA lists the DGX Station as one Blackwell Ultra GPU with 252 GB of HBM3e at 7.1 TB/s, a 72-core Grace CPU with 496 GB of LPDDR5X at 396 GB/s, and NVLink-C2C at 900 GB/s. The coherent pool is 748 GB. The fast part is the 252 GB. A 405B model at Q4 is about 243 GB after the 1.2, near the top of HBM. A 1 trillion parameter model at Q4 is about 600 GB after the 1.2. NVIDIA says the Station supports models up to 1 trillion parameters. That estimate fits the 748 GB pool and extends into the slower LPDDR5X, so speed depends on which tensors stay in HBM.
When each new token rereads the weights, tokens per second cannot exceed bandwidth divided by bytes reread. Spark's guide lists 273 GB/s, so a 4 GB file has a ceiling near 70 tokens per second.
The Station page lists 1,600 W and a 20 A circuit, one PCIe Gen5 x16 slot, and two slots that are x16 in shape and x8 electrically. Spark uses a 240 W supply. llama.cpp runs on NVIDIA with CUDA, on AMD with HIP, and on Apple silicon with Metal. vLLM serves a model that already fits, with PagedAttention for the cache.
Fine-tuning costs more. EleutherAI adds gradients, Adam state, and activations. Their AdamW line is 12 bytes per parameter beside the training weights. Read the Hugging Face model card before commercial use, and keep other people's personal data off a shared machine unless you have a lawful basis under the EU GDPR.
Fit 405B on Station memory and estimate a ceiling
DGX Station memory is 252 GB HBM3e at 7.1 TB/s plus 496 GB LPDDR5X at 396 GB/s, 748 GB coherent in total. For a 405 billion parameter model at Q4 (0.5 byte per weight, then times 1.2), say whether the weights fit in the 252 GB HBM slice alone. Then compute a decode ceiling in tokens per second if each token rereads the raw Q4 weights (parameter count times 0.5) from HBM at 7.1 TB/s. Explain, in one short paragraph, why that ceiling falls if weights spill into LPDDR5X at 396 GB/s. Keep fine-tuning separate: add EleutherAI's 12 bytes per parameter for AdamW and say how that compares with the inference total.
Numbers to keep beside the builder
| Piece | Figure used on this page |
|---|---|
| FP16 / Q8 / Q4 | 2 bytes, 1 byte, 0.5 byte per weight |
| Running a model | weight bytes × about 1.2, then add the context cache |
| Laptop tier | about 16 GB, with room left for the operating system |
| Home PC tier | a graphics card with 24–32 GB |
| DGX Spark | 128 GB unified LPDDR5x, 273 GB/s; NVIDIA documents models up to 200 billion parameters on one unit, and up to 405 billion on two linked units |
| DGX Station | 748 GB coherent memory: 252 GB HBM3e at 7.1 TB/s plus 496 GB LPDDR5X at 396 GB/s; NVIDIA documents models up to 1 trillion parameters; 1,600 W and a 20 A circuit |
| Fine-tune | weights plus gradients, optimizer state, and activations; AdamW is 12 bytes per parameter in the EleutherAI breakdown |
Treat a Q4 result that lands within about 10 percent of a tier as a maybe. Real GGUF files, the KV cache, and the operating system all sit on top of the 1.2 rule.
What this changes before you pay
The tier tells you where your money goes. Laptop or Home PC means a model in that class can run on a machine you may already have, so the continuing cost is electricity and your time. Spark or Station means hardware past a typical home graphics card. The other way to run that model is a hosted API, billed per token on the provider's own price page.
NVIDIA's Station page tells buyers to order from partner manufacturers. This page quotes the published hardware figures and leaves the price off until a sourced list price exists. If the station is bought, it can be checked part by part against the published spec above once shared build lists open. Until then, those NVIDIA figures are the reference.
A shared community DGX Station would run the models whose estimate sits above a home graphics card, on the published 748 GB coherent pool, as one machine for the community. About the shared station
Sources
- NVIDIA DGX StationOfficial spec: 252 GB HBM3e, 496 GB LPDDR5X, 748 GB coherent memory, up to 1 trillion parameters, 1,600 W, 20 A circuit, and the PCIe slot layout.
- DGX Spark hardware overviewOfficial guide: 128 GB unified LPDDR5x, 273 GB/s, 240 W supply, models up to 200 billion parameters, or 405 billion on two units.
- Hugging Face model memory estimatorWeight-only memory by data type. States that the tool measures loading, and that inference can add up to about 20 percent, citing EleutherAI.
- EleutherAI Transformer Math 101Bytes per weight for int8, fp16, and fp32, the 1.2 inference heuristic, and the AdamW optimizer line of 12 bytes per parameter. Summarized here; the post is theirs.
- llama.cppMIT-licensed local inference, including 4-bit and 8-bit quantization, plus CUDA (NVIDIA), HIP (AMD), and Metal (Apple) backends.
- vLLM documentationServing library for models that already fit in memory, including PagedAttention for the key-value cache and an OpenAI-compatible API.
- Hugging Face model hubModel cards with parameter counts, file sizes, and the license you need to read before commercial use.
- Ollama model libraryReady-to-run local model packages, useful once the builder says a Q4 model fits the machine you have.