Large context windows are tempting. Loading every document at once cost more and picked up outdated rules.
A thought to take with youA large context window does not replace choosing the right information. Select first, then send.
What we expected
With models that accept very long inputs, we tried skipping retrieval altogether and giving the model the full set of documents for every question.
What happened
Costs went up sharply and answers became less reliable. The model sometimes relied on an older clause near the start of a document instead of the amendment that replaced it further on.
What we changed
We went back to selecting a small amount of relevant text, filtered by date, status and access before the model sees it. Smaller, cleaner context gave better answers at a fraction of the cost.