Open & local models
An open model is a weight file you can download and run on your own computer when the file fits in memory and the license allows that use. This page shows how to read a size such as 7B or 70B, check the license, and choose a fast reply or a model that writes its reasoning first.
Beginner
What you are downloading
A base model is a file of learned numbers, called weights, plus a short header that says how to read them. The B in 7B or 70B means billions of parameters, one stored number each. At 16-bit precision each number takes 2 bytes, so a 7-billion-parameter file is about 14 GB and a 70-billion-parameter file is about 140 GB, before memory for the conversation. Quantization stores each number in fewer bits. llama.cpp, a program that runs models on your own computer, supports integer sizes from about 1.5 bits through 8 bits.
Open on a download page means you can fetch the weights. The license says what you may do next. Gemma 4, from Google DeepMind, uses the Apache License 2.0. You may use, change, and distribute it, including in a product, if you keep the license, mark files you changed, and leave the original notices in place. It includes a patent grant from contributors and disclaims warranty. A downloadable model can still ban commercial use or add an acceptable-use list. Read the license on the model card before a client project.
A reasoning model writes its steps, then a shorter answer. On Gemma 4, thinking starts when the system prompt begins with <|think|>. Those steps take time and fill the context window, so a short question can use the faster mode with thinking left off. If a local run fails, check three causes: memory filled and the process died, the file on disk differs from the card, or the prompt was missing the error message and the code around it.
Will this model fit my computer?
I have [RAM] GB of memory and [graphics memory] GB of graphics memory. I want to run [model name]. Explain what a weight file is. Estimate whether a 7-billion-parameter model and a 70-billion-parameter model would fit at 16-bit (2 bytes per number) and at about 4-bit, before conversation memory. Read the license on the model card and say, in plain language, what I may do with the weights, including commercial use. If the program exits while loading, give me the first three checks. Do not assume the model already fits.
Intermediate
Read the card, then load a file
Read the model card before the download. It lists parameter count, context window, license, and accepted inputs. On the Gemma 4 card, E2B and E4B list 128,000 tokens. The 12B, 26B A4B, and 31B models list 256,000. E2B counts 2.3 billion effective parameters and 5.1 billion with embeddings. The 26B A4B mixture-of-experts model stores about 25.2 billion parameters and uses about 3.8 billion on each token, a speed the card compares with a model near 4 billion. The dense 31B model uses all 30.7 billion on every token. Every expert still sits in the file, so disk size follows the total and tokens per second follow the active count.
GGUF is the single-file format llama.cpp loads. Names carry a size label and an encoding such as Q4_K_M or F16. A 4-bit file is on the order of a quarter of the 16-bit size. Quality drops first on exact code, long arithmetic, and rare words. The conversation cache grows with the tokens you keep, so a model that loads for a short chat can die on a long prompt.
The llama.cpp README uses llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF to fetch a small public file and open a chat, and llama serve for a local API. A kill during load means the file exceeds free memory. If the answer misses your function completely, the prompt may have been cut, or thinking was on for a one-line fix. Record model id, encoding, context length, and the command.
Plan one repeatable local run
Here is a model card: [paste the card] Tell me the parameter count, the context window, the license, and whether the model is dense or a mixture of experts. If both a total and an active parameter count are listed, say which one drives file size and which one drives tokens per second. Estimate the 16-bit weight-file size at 2 bytes per parameter, and a 4-bit planning size at about a quarter of that. Name the memory used on top of the file. Then give the steps to prove llama.cpp with a tiny public GGUF before I download this model, and list the four fields I should write down so a later run matches this one.
Expert
Latency, diffs, and a job queue
llama.cpp runs one model on one machine, on CPU or GPU (CUDA, Metal, or Vulkan), and can split leftover weights into system memory. vLLM serves many requests with the model resident on the GPU. Its offline API generates, chats, and enqueues batches with continuous batching. Use llama.cpp for a fit test and vLLM for a shared queue.
Time to the first visible token and total answer time both rise with thinking on, because the model writes its reasoning first. Gemma 4 enables that with <|think|> at the start of the system prompt. Keep earlier thoughts out of later turns, except on a tool call. The key-value cache grows with context length and with parallel jobs, so cap each job. Load 4-bit weights when 16-bit will not fit, and 8-bit or 16-bit for a precise code edit.
For a review, pass the git diff, the failing test, and a rule that untouched files stay as they are. Ask which handle leaks or which list grows without a bound, then run the tests yourself. HumanEval scores short functions by unit tests. Its repository is unmaintained and leaves execution commented out so model code runs only in a sandbox. SWE-bench scores real GitHub issues from 12 Python repositories, 2,294 full and 500 Verified, as percent resolved.
The EleutherAI harness repeats a published task through llama.cpp or vLLM. Log date, machine, quant, context cap, and harness version with the score. A shared DGX Station could work as a one-job queue for a higher-precision 70B model or a full harness pass.
Review a diff and spec the run log
I will paste a git diff and a failing test. Do not rewrite files the diff does not touch. Name the handle that is never closed or the structure that grows without a bound, then propose the smallest patch. Wait for me to run the tests. Separately, specify how to record time to first token, tokens per second, and resident memory at load, at first token, and after exit. Say whether this job should run in llama.cpp on a single machine or as one job on a shared NVIDIA DGX Station (748 GB coherent memory). List the fields to store beside an EleutherAI lm-evaluation-harness score: date, machine, quant, context cap, and harness version. Diff: [paste] Failing test: [paste]
Will the file fit?
These gigabyte figures are arithmetic from the parameter count: 2 bytes each at 16-bit, and about a quarter of that at 4-bit. A real GGUF file is often a little larger because some layers stay at higher precision, and the conversation cache is extra. Measure the file and the free memory on your machine.
| Class | 16-bit weights | 4-bit, planning figure | Where it usually belongs |
|---|---|---|---|
| 7B | about 14 GB | about 4 GB | Many computers with 8–16 GB of graphics memory, if the context stays short |
| Gemma 4 31B (30.7B on the card) | about 61 GB | about 15 GB | A workstation GPU or a large unified-memory machine |
| 70B | about 140 GB | about 35 GB | A high-memory workstation. A long context makes a single consumer graphics card tight |
| NVIDIA DGX Station | 748 GB of coherent memory: 252 GB of HBM3e beside the GPU and 496 GB of LPDDR5X beside the CPU | The same pool, at higher precision and longer context | A DGX Station-class deskside machine, if Polaris buys one. NVIDIA states support for models up to about 1 trillion parameters |
If 16-bit will not load, try 8-bit before 4-bit when the task is code. If 4-bit loads and then dies mid-chat, lower the context length before you discard the file.
What a shared run will record
If Polaris buys a shared machine, a heavy run there should be one queue item: model id, encoding, context cap, task (chat, diff review, or harness), and a time limit. The result row should carry the date, the machine name, and the command. One job at a time is the right default, because a second model loaded beside the first splits the same memory pool. The market and the job board are not open yet, and there is no shared queue to join today. You can already keep that row on your own computer.
Checks before you trust a local answer
- Match the filename to the card: parameter class and encoding.
- Read free memory before load, and again after the process exits.
- Start at a short context. Raise it only after a successful reply.
- Leave thinking off for a syntax error. Turn it on when the bug crosses files.
- Run the tests yourself before you trust a reply that says they passed.
- For benchmark code, use a sandbox. The HumanEval repository ships with execution commented out until you accept that risk.
- Keep the license file next to the weights if you redistribute them. Apache-2.0 requires the license, change notices, and the original attributions.
A shared community DGX Station, with the 748 GB of coherent memory NVIDIA lists for that system, would be a one-job queue for open models and harness runs that do not fit on a laptop. About the shared station
Sources
- NVIDIA DGX StationOfficial spec page: 748 GB coherent memory (252 GB HBM3e plus 496 GB LPDDR5X), and NVIDIA's statement that the system supports models up to 1 trillion parameters.
- llama.cppLocal inference in C/C++, integer quantization from about 1.5-bit through 8-bit, and the README commands for a small GGUF chat and a local server.
- Gemma 4 model cardSizes, context lengths, mixture-of-experts counts, thinking control, and the Apache-2.0 statement. Page text is under Creative Commons Attribution 4.0. This page paraphrases it.
- Apache License, Version 2.0The license text: use, modification, distribution, patent grant, notice requirements, and the warranty disclaimer.
- vLLM offline inferenceCurrent stable docs for LLM.generate, LLM.chat, and the enqueue APIs, including how batched serving keeps a GPU busy.
- Language Model Evaluation HarnessEleutherAI's framework for repeating published tasks on a local file, including a llama.cpp server and a vLLM backend.
- SWE-benchLeaderboard for repository repairs: 2,294 issues from 12 Python repositories in the full set, 500 in the human-filtered Verified set, scored as percent resolved.
- HumanEvalThe original unit-test harness for short coding tasks. The repository is unmaintained and tells you to execute model-written code only in a sandbox.