Compute & power

Compute is the chip, the memory, and the electricity that turn a model into text or images. Power is the electricity that work draws, and it shows up on your bill. This page shows how to price a month of use in kronor, how to tell whether a model fits in the computer you have, and how to read a hardware result that was measured on a named machine.

Beginner

What a chat costs, and whether the model fits

A cloud service usually charges per token or per image. A token is a piece of text, a short word or part of a longer word. Swedish and English can split one sentence into different token counts, so take the count from the provider's tokenizer or from the invoice. Divide a month of tokens by 1,000,000 when the price is per million, multiply by that price, then convert to kronor at today's exchange rate. Price images as the listed price times the image count. Add any monthly fee, and include retries.

A model on a computer you own has no per-token charge. You pay for the machine and for the electricity it draws. For light chat, use a small model on the laptop already on your desk. If the weights are larger than the memory on that machine, the model will not load at that precision.

Gigabytes of weights are the parameter count in billions times the bytes per value: 4 for FP32, 2 for FP16 or BF16, 1 for 8-bit, and 0.5 for 4-bit. Seventy billion parameters is about 140 GB at 16-bit and about 35 GB at 4-bit. Hugging Face's guide, citing EleutherAI, says to allow up to about 20 percent more while the model answers. A long thread stores a cache of earlier tokens on top of that.

Check the Hugging Face memory calculator against the product-page memory before you buy. NVIDIA lists the DGX Station at 1,600 watts, a 20-amp circuit, 252 GB of fast GPU memory, and 748 GB coherent memory.

Price this month in kronor

Help me estimate one month of AI use in Swedish kronor. I will paste a provider price page (per million tokens, and per image if any) and a short log: number of chats, rough length, number of images, and any monthly fee. Convert tokens using only the price I paste. Ask me for today's SEK exchange rate if I forgot it. Then estimate a local alternative from the parameter count, the precision (FP16, FP8, or 4-bit), and the graphics-memory size in GB that I give you. Use 4, 2, 1, or 0.5 bytes per parameter for FP32, FP16/BF16, 8-bit, and 4-bit. Add 20 percent for inference overhead, and say that a long chat needs still more memory for the cache. End with one recommendation: stay on the cloud, use a smaller local model, or this precision does not fit this machine. Leave any price I did not paste as a blank.

Intermediate

Fast memory, slow memory, and a published test row

For each new token, the model reads all of its weights again. On a small batch that read often takes longer than the math, so memory bandwidth sets the pace. Kipply's inference notes show how a large batch can move the limit onto compute.

NVIDIA lists 7.1 TB/s on 252 GB of HBM3e and 396 GB/s on 496 GB of LPDDR5X, joined by NVLink-C2C at 900 GB/s into up to 748 GB of coherent memory. The fast pool shares its space with the conversation cache. A model that spills into the slower pool can still load, and then each new token waits on that slower read.

The page lists FP4 peaks of 20 PFLOPS with sparsity and 15 PFLOPS without, FP8/FP6 at 10 PFLOPS, and FP16/BF16 at 5 PFLOPS. Those are peak figures, and a real job runs well below them. Seventy billion parameters take about 140 GB at 16-bit, inside the fast pool, and about 280 GB at 32-bit, which uses the slower memory too. A trillion parameters at 16-bit is about 2,000 GB, past the 748 GB NVIDIA specifies, so that size needs a smaller number format and uses the slow pool.

There is no shared queue to join yet. If a community machine is bought, each test would be published as one row: date, machine name, model, precision, prompt length, tokens per second, and watts. In MLPerf Inference v6.1, compare only rows that share a model, a scenario, and a quality target, then read the change log on the datacenter page.

Will this model sit in fast memory?

Decide whether an open model fits in fast GPU memory or spills into slower memory. I will give the parameter count, the precision, the context length I care about, and the machine: a laptop with a stated GB of graphics memory, or a DGX Station with 252 GB HBM3e at 7.1 TB/s and 496 GB LPDDR5X at 396 GB/s inside 748 GB coherent memory. Estimate weight memory as parameters times bytes (2 for FP16/BF16, 1 for FP8 or INT8, 0.5 for 4-bit). Add 20 percent for inference overhead. Say the KV cache grows with context, and compute a cache size only if I also give layers, KV heads, head size, and bytes per cache element. Then say whether the weights fit in 252 GB, only in the full 748 GB, or in neither. Keep any peak PFLOPS figure separate from tokens per second. Ask me for any missing number.

Expert

Watt-hours, precision, and the monthly bill

Record average watts and seconds for one fixed prompt. Watt-hours equal watts times seconds, divided by 3,600. Per 1,000 tokens, divide those watt-hours by the output tokens and multiply by 1,000. Note precision, batch, context length, other jobs on the box, and whether watts came from a wall meter or from software. An hour at the 1,600 W nameplate is 1.6 kWh. Repeat the prompt on a consumer card and a laptop.

FP8 stores one byte per value and FP16 or BF16 stores two, which halves weight memory and the bytes moved per token. NVIDIA's article splits FP8 into E4M3, a tighter range, and E5M2, a wider range for gradients. Blackwell adds MXFP8, one scale per block of 32 values, and FP4. Compact formats can clip outliers, so keep a quality score beside the speed. ConnectX-8 can join two stations at up to 800 Gb/s. Splitting a model across that link adds traffic on every step, so a second station would only be needed to hold a larger model. MIG can split the chip into up to seven slices, each with its own memory, so a long job can run in one slice without slowing the rest. Write training checkpoints on a timer so a stop can resume.

Monthly cost on your own machine is the seller's quote divided by the months you will keep it, plus measured kilowatt-hours times the kronor per kilowatt-hour on the bill. Include grid fees and tax for the amount on the bill. Get the purchase price from the seller. The cloud side is the invoice, or tokens times that day's price, plus rented hours that sat idle.

Build a watt-hour and monthly-cost row

Turn my notes into one comparison row for a local machine versus a cloud API. I will paste: model name, precision, context length, batch size, output tokens, seconds, average watts, whether the reading was a wall meter or a software sensor, the machine name, the date, my electricity price in kronor per kWh and whether grid fees and tax are included, the hardware purchase price, how many months I will spread that price over, and the cloud price or invoice line for the same prompts. Compute watt-hours per 1,000 tokens as (average watts × seconds / 3600) / (output tokens / 1000). Compute local monthly cost as purchase/months plus kWh × price. Compute cloud cost only from numbers I paste. If I used the 1,600 W DGX Station nameplate in place of a measurement, label the energy figure as a ceiling. Leave tokens per second, a MIG plan, and any second-machine speedup blank unless I measured them. List any field still missing.

Weight memory at a glance

Weights in gigabytes are about the parameter count, in billions, times the bytes per parameter. Allow up to about 20 percent more for a short answer, then more again for the conversation cache. The rows below are that arithmetic applied to round parameter counts.

Precision	Bytes per parameter	8 billion	70 billion	1 trillion
FP32	4	32 GB	280 GB	4,000 GB
FP16 or BF16	2	16 GB	140 GB	2,000 GB
FP8 or INT8	1	8 GB	70 GB	1,000 GB
4-bit	0.5	4 GB	35 GB	500 GB

On the DGX Station spec the fast pool is 252 GB of HBM3e. A 70-billion-parameter model in 16-bit weights lands inside it. The 32-bit copy is about 280 GB and needs the slower memory as well. A 1-trillion-parameter model at 4-bit is about 500 GB of weights: inside the 748 GB coherent total, and outside the fast pool. At 8-bit those weights are about 1,000 GB, which is past the whole machine. Fitting a 1-trillion-parameter model inside the 748 GB NVIDIA specifies depends on a compact number format.

What a public test row should contain

No community machine is publishing rows yet. When one does, keep each run to fields a reader can audit:

Date and machine name, so a later run on different hardware stays on its own line.
Model, revision, and precision, including the FP8 recipe or the 4-bit method.
Context length, batch size, output tokens, and tokens per second.
Average watts, the meter you used, and watt-hours per 1,000 tokens.
A quality check beside the speed, using one fixed prompt scored the same way each time.
For an MLPerf number quoted from elsewhere, the scenario name, a link to that result row, and a look at the change log.

A shared DGX Station would make the machine-name field point at one named machine whose runs are published, with MIG slices when several jobs run together and a checkpoint on disk when a job has to stop.

Monthly cost checklist

Write down the hardware quote from the seller. Divide it by the number of months you will keep the machine.
Measure kilowatt-hours for a normal week. Multiply by the kronor per kilowatt-hour on your bill. The 1,600 W rating is a ceiling: 1.6 kWh if the system held that draw for a full hour.
Add any software you pay for. Add disks and the monitor if you want the whole desk on the same line.
On the cloud side, use the invoice or the current price per million tokens and per image. Include idle time if you rent a machine by the hour.
Compare the two monthly totals on the same prompts. For small models, the laptop you already have may stay the cheaper choice. A station would be for 70-billion-parameter models that need fast memory.

A shared community DGX Station would let members run models that need the 252 GB of fast memory, and publish each run with the date and the machine name so the watt and speed figures on this page come from a box they can point to. About the shared station

Sources

NVIDIA DGX StationSpecification table: 748 GB coherent memory, 252 GB HBM3e at 7.1 TB/s, 496 GB LPDDR5X at 396 GB/s, NVLink-C2C at 900 GB/s, tensor-core peaks, 1,600 W on a 20 A circuit, up to seven MIG instances, and support for models up to 1 trillion parameters.
NVIDIA DGX Station Development GuideWhat the deskside GB300 system is for: local inference and fine-tuning, the coherent memory split, and linking two stations with ConnectX-8.
Hugging Face model memory estimatorThe accelerate estimate-memory command. The guide says the number is for loading the weights, and that inference can add up to about 20 percent.
Hugging Face model memory calculatorInteractive calculator for a model on the Hub at float32, float16, int8, and int4. Use it before comparing a laptop or a graphics card.
EleutherAI, Transformer Math 101Bytes per parameter for INT8, FP16/BF16, and FP32, and the experience-based rule that inference overhead stays within about 20 percent of the weights. Cite the post if you reuse the formulas.
kipply, Transformer Inference ArithmeticWhy a small batch is often limited by memory bandwidth, how the KV cache grows with context, and what it costs to split a model across chips.
NVIDIA technical blog, Floating-Point 8E4M3 and E5M2, why fewer bits cut memory traffic, and how Blackwell MXFP8 (a scale per 32 values) and FP4 extend the same idea.
MLPerf Inference v6.1 resultsThe published v6.1 inference results, including newer agentic tests. Read each number together with the datacenter suite page, which explains scenarios, quality targets, and the change log: https://mlcommons.org/benchmarks/inference-datacenter/