01 / THE QUESTION
Can a model use the context it accepts?
A request fitting inside a context window does not establish that the model can use all of it. This proposed study separates accepting a long input from producing a correct, grounded answer.
02 / SETUP
Make the conditions explicit.
Fix the model and its exact revision, tokenizer, serving runtime, GPU, cache settings and generation settings. Record the context allowance and reserve the same output budget for every comparable case.
03 / PROPOSED METHOD
A repeatable path.
- Create fixed documents with facts placed near the beginning, middle and end. Keep a reference answer for every question.
- Repeat comparable cases at increasing input lengths. Include questions that need evidence from more than one section.
- Include missing-information controls. A correct response should acknowledge when the answer is absent.
- Preserve the full outputs and score correctness separately from first-token delay, generation speed and memory use.
04 / WHAT TO MEASURE
More than a single number.
- Answer correctness and source grounding
- Recall by fact position and input length
- Time to first token and output tokens per second
- Peak memory and request failures
05 / LIMITATIONS
What this would not prove.
A small set of synthetic documents cannot establish performance on every real workload. Exact-match recall, multi-step reasoning and code work need separate reporting. Changing the model, quantization or runtime invalidates direct comparisons.
06 / NEXT STEP
Before collecting evidence.
Freeze the evaluation cases and reference answers before collecting a publishable result.
Download draft protocol ↓