Precision & memory
A model answers from the text that fits in one request, and from weights stored at a chosen precision. This page shows how a long chat, a PDF upload, or a smaller model file can change the answer, and how to measure speed and power on a long run.
Beginner
Why a long chat loses the detail
Each new reply is computed from one request that holds your instructions, any file you attached, and the earlier turns that still fit. That limit is the context window. The model measures the request in tokens, pieces of text that are often a word or part of a word. When the next turn no longer fits, the chat product drops or shortens the oldest turns. A name, a date, or a condition from the start of the chat can be missing from the request that produces the later answer.
Where a fact sits in the request changes the result. Lost in the Middle tested models that had to use one relevant passage hidden among many. Performance was often highest with the passage at the beginning or the end, and lower with the passage in the middle, including on models built to accept long inputs.
Uploading a PDF adds another cut. The product turns pages into text. Scans, side-by-side columns, and tables can lose cells, headers, and footnotes. Many products then search that text and copy only a few passages into the request. That pattern is retrieval-augmented generation: the written answer follows the passages that search returned. Ask one precise question twice. First paste only the paragraph that holds the answer. Then send the whole file. Read both replies against the page. Where they split, the file view or the search step left something out.
Those paired checks can be posted in the forum, which is open to invited members during the trial.
Check what the model can still see
I am testing what you can still use. Fact to remember: [paste one precise fact, with a name and a number]. Filler: [paste several unrelated paragraphs]. Question: [ask only for that fact]. After your answer, quote the fact as you received it. If any part of it is missing from your view of the conversation, say which part.
Intermediate
Keep the line you need
Start a new chat when the job changes. Copy across the names, dates, totals, and decisions the new job still needs, and leave the old thread. If the product adds a summary of earlier turns, read the summary before you trust the next answer. A summary often keeps the topic and drops the exception.
For a PDF, ask for a quotation before a conclusion. The quotation should be a sentence you can find in the file, with the page number when the PDF has pages. A reply that cannot show the sentence is unfinished, so open that page yourself. With a long report, send one labeled section, the part that can contain the answer, and also send the full file in a separate run of the same question. Treat the short section as the control. Note a changed year, a rounded figure, or a condition that appears in one reply only.
A smaller model file is a third way to lose detail. Quantization stores the weights, the learned numbers inside the model, with fewer bits so the file uses less memory. llama.cpp implements integer quantization from 1.5-bit through 8-bit. Weights stored at 4 bits take about a quarter of the space of 16-bit weights, plus a small overhead for the scale stored with each block. On exact names, sums, and long instructions, run the identical prompt on the small file and on an 8-bit or 16-bit file. When a fact you can check in the source changes, use the higher-precision file for that job.
Compare a short excerpt with the full file
Question: [one question whose answer is a specific figure, date, or condition]. Part 1. Use only the excerpt below. Quote the sentence you used, and give the page or heading if one is shown. Excerpt: [paste the paragraph that contains the answer]. Part 2. Answer the same question from the full PDF I upload. Quote the sentence you used. If the two answers differ, list the difference. Keep them separate, and leave the final choice to me.
Expert
Quantize, cache, and measure the run
Write one row per run: date, machine, model, file name, quantization, sampling temperature, context length, prompt, and score. Set sampling temperature to 0, or the lowest value offered.
In llama.cpp, Q4_K_S, Q4_K_M, Q5_K_M, and Q8_0 name the packing. S, M, and L change how much of the file uses the lower bit width, with a scale stored beside each K-quant block. Quantize once from 16-bit or 32-bit weights. Use a wider file for a figure, a citation, or a clause, and when a paired run changes a fact you can check.
Prompt caching on the OpenAI API stores the key-value state of a prefix that matches exactly, including tools and instructions at the front. Put stable text first and the changing question last.
Batching runs many sequences on the same weights, while each sequence keeps a key-value state that grows with length. Tokens per second for the machine can rise as one person waits longer for a first token. Log both.
Put one sentence the file lacks near 10, 50, and 90 percent of a public annual report or agency PDF. Ask for that sentence, then ask a question that needs two places, and score the tasks apart. Compare embeddings on MTEB by task and language, pipelines on RAGBench, and indexes on ANN-Benchmarks.
On a long run, log chip temperature, wall-plug watts, and tokens per second every few minutes. MLPerf Inference: Datacenter reports wall-plug system power for Server and Offline, and energy per stream for single-stream and multi-stream. A hot machine can slow down after the first minute.
One row for a precision test
Treat this as a measurement. Set sampling temperature to 0, or to the lowest value you offer. State the model, the file name, the quantization label, and the context length you used. Document: [paste or attach a public annual report or agency PDF]. Needle, a sentence that is absent from the original: "[write one unique sentence]". Assume it sits near the middle of the document. Task A: Repeat the needle sentence exactly. Task B: [ask a question that needs two separated places in the document]. Quote both places. Reply in a table: exact match on Task A (yes or no), the two quotes for Task B, and any number you could not find. Include only what the document contains.
A paired run you can repeat
Use the same wording for every run in a pair.
- Record the product or local runner, the model name, and the file name, including a quant label such as Q4_K_M or Q8_0.
- Set sampling temperature to 0, or the lowest value offered, and write that number down.
- Ask one question with an answer you can point to in the source.
- Run A on the short excerpt or the higher-precision file. Run B on the full upload or the smaller file.
- Require a quotation and a page or section for each run.
- Mark a miss when a year, a total, a name, or a condition differs, or when the quotation is absent from the file.
- For a needle, record document length, depth in percent, exact match, and the result of a second question that needs two places.
- For a long run, add the clock time, the chip temperature, watts at the wall, and tokens per second at several points along the run.
What changed, and what to log
| What you see | What to log next |
|---|---|
| An early detail disappears after many turns | The token count of the current request, and whether older turns were shortened or summarized |
| The full PDF and a short excerpt disagree | Whether each reply quotes a sentence that is really in the file, and whether search returned that page |
| The smaller file changes a checkable fact | The same prompt on a wider file, quantized once from 16-bit or 32-bit weights |
| A cached call is faster and still misses a fact | Whether the unchanged prefix contains that fact |
| Tokens per second fall after the opening minutes | Chip temperature and wall-plug watts across the whole run |
When to leave the smallest file
Move to a wider file, such as 8-bit or 16-bit, in these cases.
- The answer is a figure, a name, a date, or a clause that someone else will rely on.
- A paired run already changed a fact you can find in the source.
- The document is long and the lines you need sit in the middle of it.
- You are publishing the row for other people to reuse.
A very small file still fits a private draft. Repeat the checkable questions on the wider file before you keep the result.
A shared DGX Station would give the community one known machine with the memory to repeat the same prompt on a large open model at a higher-precision file and at a smaller file, and to publish the date, the machine name, and the tokens per second. About the shared station
Sources
- llama.cppLocal inference in C++, with integer quantization from 1.5-bit through 8-bit. The quantize tool accepts an imatrix file so weights that affect the output more can keep extra precision. MIT license.
- Lost in the Middle: How Language Models Use Long ContextsNelson F. Liu and colleagues show that finding a relevant passage is often stronger at the start or end of a long input than in the middle. arXiv non-exclusive distribution license.
- Needle in a HaystackA test runner that scores retrieval across context length and needle depth, including multi-fact and multi-hop tasks. MIT license. Attribute the project if you reuse the code.
- MTEB leaderboardLive results for embedding models. The benchmark paper (https://arxiv.org/abs/2210.07316) found that no single method led every task, so compare by task and by language.
- RAGBenchCode for a retrieval-augmented generation benchmark. The paper (https://arxiv.org/abs/2407.11005, CC BY 4.0) describes 100,000 labeled examples across industry documents and the TRACe metrics.
- MLPerf Inference: DatacenterStandard scenarios for inference speed. Server and Offline report system power at the wall. Single-stream and multi-stream report energy per stream.
- OpenAI prompt cachingHow an unchanged prefix reuses its key-value state. Cache lifetime, minimum length, and price depend on the model, so check this page before you budget a run.
- ANN-BenchmarksRecall against queries per second for approximate nearest-neighbor search, including several vector-database algorithms. The plots measure the index, separate from the embedding model.