Reading a few answers by eye stops working the moment something changes. How we turn review into a repeatable check.
A thought to take with youWrite down what a good answer must do, collect real examples, and re-run them every time the model, prompt or data changes.
Why checking by eye fails
Early on, everyone reads a few answers and agrees they look fine. Then a model is updated, a prompt is edited or a document is added, and quality shifts quietly. Nobody re-reads the old examples, so nobody notices until a customer or colleague does.
What we check instead
We keep a set of real questions with the answers a domain expert expects, including awkward ones the system should decline. Each run checks a few plain things: is the format right, does the answer cite the right source, and does it refuse when it should.
The set runs automatically on every change, so a drop in quality shows up before it reaches people.