Apps & interfaces
The app is the window around a model: a website, a program on your own computer, or a shared page that can also open your files. Pick the window for the job, then check where the text goes and what the program is allowed to read.
Beginner
Three ways to talk to a model
An app is the window you type into. The model is the file of learned weights that writes the reply. Three app types cover a first setup.
A browser chat runs on the provider's computers. The prompt leaves your machine, you install nothing, and you need a network.
A local window runs the model on your computer and keeps working offline after the download. Ollama installs on macOS, Windows, and Linux. ollama run gemma4 downloads a model and chats in the terminal, and other apps can call it at http://localhost:11434. llama.cpp, the engine under many of these tools, chats from its own command, and llama serve adds a page on that machine.
A file-handling agent may also open folders or call tools. Open WebUI is a self-hosted browser interface for Ollama and for servers that speak the OpenAI chat API, a shared format for chat requests. It attaches documents and runs tools. Grant one folder.
Weights sit in memory while the model runs. A parameter is one number in that file. Video memory (VRAM) is fast and limited. System memory (RAM) is larger and slower. If the file exceeds VRAM, llama.cpp keeps some layers on the graphics card and the rest in RAM, slower than a full fit in VRAM. Quantization stores each weight in fewer bits. llama.cpp documents widths from 1.5-bit through 8-bit. A laptop runs a small quantized model. Models around 70 billion parameters need much more memory.
The market and the job board open later. Use the checklist below before you let an app see your files.
Choose a first app for this computer
I want to try an AI chat and I may want it to run on my own computer. Ask me my operating system, how much memory I have, and whether I have a separate graphics card. Then choose one first step among a browser chat, Ollama, and llama.cpp. Say what that app can read on my disk, give the install page or the first command, and name the setting that would send my text to a cloud server.
Intermediate
Interface, server, and where the text sits
Keep three layers apart. The interface is the screen. The server holds the weights and answers requests. The model file is the weights. Ollama includes a plain chat with its server. Open WebUI supplies the interface, so you can point it at Ollama now and at another engine later.
A local Ollama API listens at http://localhost:11434, with OpenAI-compatible routes under /v1. Those calls use a downloaded model and send no API key. The hosted API at https://ollama.com requires a key and sends the prompt to Ollama's cloud. An online-only model, or a cloud sign-in, moves text off the computer. Check that control before you paste notes from work.
Open WebUI stores chats in a database on the computer that runs it: SQLite by default, or PostgreSQL. Files stay on local disk or move to a cloud bucket you configure. Retrieval, called RAG (retrieval-augmented generation), copies documents into that store and splits them into passages the model can search, so the app holds its own copies. Web search can send the query to an outside service. Leave it off when the question should stay local.
One computer runs the server. Others on the same network open the page. Put a password on it. For each test, record the app version, the exact model name, and whether the run was local or hosted. A speed figure from another machine only describes that hardware, so write down your own figure with the date.
Trace one message before you paste a document
I run a local model behind a web interface such as Open WebUI, with Ollama or another OpenAI-compatible server. Trace one chat message: which program holds the weights, which address it listens on, where the chat history is stored, and which settings copy my text off this machine (cloud model, API key, web search, document upload, a cloud file bucket). End with a short check I can follow before I paste a document from work.
Expert
Servers, memory, and a row someone can repeat
Measure speed on the inference server, not in the chat window. On one desktop, learn llama.cpp or Ollama first. GPU offloading in llama.cpp puts some layers in VRAM and the rest in system RAM when the file is larger than the card.
Budget the KV cache: attention state for tokens already read. Weights stay fixed. The cache grows with context length and with how many chats run at once. vLLM uses PagedAttention and continuous batching, so a new request joins work already on the GPU. vLLM returns tool calls as structured requests the client runs. TensorRT-LLM compiles a tuned engine for one NVIDIA GPU and one model. Check its support matrix first. vLLM covers many architectures. TensorRT-LLM is built for one fixed model on NVIDIA hardware.
Log one row per run: date, machine name, engine and version, model id, quantization, context length, concurrency, and the metric with its definition. MLPerf Inference: Datacenter, from MLCommons, times a fixed model under scenarios such as Server and Offline, with a quality target. Closed entries keep the model equivalent to the reference.
The DGX Station guide specifies up to 748 GB of coherent memory: up to 252 GB of HBM3e at up to 7.1 TB/s and 496 GB of LPDDR5X at up to 396 GB/s, linked by NVLink-C2C. The guide says that pool runs models up to about 1 trillion parameters, including DeepSeek-R1, gpt-oss-120B, and a 235-billion-parameter Qwen model, and that developers can use tools such as PyTorch, vLLM, SGLang and Ollama on its Ubuntu system. Start with the graphics card you have. When the model and its cache no longer fit, a shared machine would take over, and a second station would only be needed for the very largest models.
Pick a server and a log line
I am choosing a local inference server for a shared NVIDIA test machine. Compare llama.cpp, vLLM, and TensorRT-LLM on GPU offloading, KV-cache growth, continuous batching, tool calling, and the fields to write in a results row so another person can repeat the run. The client speaks the OpenAI chat API. Mark every speed claim as a measurement I still have to take on the actual machine, and say what would change if the weights plus the KV cache no longer fit in the GPU.
What the app is allowed to touch
Check this before the first prompt that matters.
- Download the installer from the project's own site or its GitHub releases.
- On first launch, notice whether it only opens a port on your machine or also asks you to sign in.
- Grant one project folder. Access to your whole user folder can wait until a task needs it.
- Read the tool list before you enable tools. A tool can read files and run commands.
- Find where chat history is stored, and whether uploaded files stay on disk or go to a bucket.
- Treat an API key, a hosted model name, or web search as a path off the machine.
- After an update, read what the defaults changed. A new version can change what the app may reach.
Which interface does which job
| Interface | Where the model runs | How you reach it | Use it when |
|---|---|---|---|
| Browser chat | The provider's computers | The provider's website | You want a first conversation and accept the prompt leaving your machine |
| Ollama | Your computer | Terminal, and http://localhost:11434 |
You want one installer, a local chat, and a local API |
| llama.cpp | Your computer | Its command line, llama serve, or its API |
You want the engine, quantization, and a CPU/GPU split |
| Open WebUI | The computer that hosts it | A browser, aimed at Ollama or an OpenAI-compatible server | Several people share one page, with documents and tools |
| vLLM | A server, often Linux with a GPU | An OpenAI-compatible API | Many requests at once, tool calls, and a run you will log |
| TensorRT-LLM | An NVIDIA GPU | The engine you compile, then its HTTP service | The model and the NVIDIA machine will stay fixed |
How a test row will be published
Each measured run becomes its own row: date, machine name, operating system, interface, engine version, model id, quantization, context length, concurrency, and the metric with a one-line definition. Rows show up when a run has been made. MLPerf Inference: Datacenter stays the industry suite for comparing datacenter systems under shared rules. A row here records what this app did on this machine, on this date.
A shared community DGX Station would give members a queue for open-weight models that do not fit on a laptop, and each finished run would come back as one dated row carrying that machine's name. About the shared station
Sources
- llama.cppThe C/C++ inference engine: install paths, llama serve, quantization from 1.5-bit to 8-bit, and CPU plus GPU hybrid runs when a model is larger than VRAM.
- OllamaInstallers for macOS, Windows, and Linux, the local chat command, and the local API used by desktop and web interfaces.
- Ollama API documentationLocal base URL http://localhost:11434 versus the hosted API at https://ollama.com, including which calls need an API key.
- Open WebUISelf-hosted chat interface for Ollama and OpenAI-compatible servers, with tools, document retrieval, and SQLite or PostgreSQL storage.
- vLLMServing engine with PagedAttention, continuous batching, an OpenAI-compatible API, and tool-calling parsers.
- NVIDIA TensorRT-LLMNVIDIA's library for building an optimized inference engine and a runtime for large language models on NVIDIA GPUs, plus the support matrix.
- NVIDIA DGX Station Development GuideOfficial description of the deskside Grace Blackwell system: up to 748 GB coherent memory, model sizes up to about 1 trillion parameters, and software such as vLLM and Ollama that runs on it.
- MLPerf Inference: DatacenterMLCommons benchmark for how fast a system runs a fixed trained model, with scenarios, quality targets, and published results.