evalgate
In DevelopmentA CI gate for RAG and LLM apps: a number that moves when answer quality breaks
Role: Solo developer
Solo · Since Nov 2026 · Public repo
A versioned eval set, a committed baseline and a command that exits non-zero when quality drops past a tolerance. It runs with no API keys and gives the same result every time, so it can run on every pull request.
Problem
Prompt and retriever changes tend to merge with "looks fine on the three examples I tried". I wanted the same check for answer quality that a test suite gives for code.
Solution
The repo ships a small reference RAG app (an original 25-passage corpus, a stdlib BM25 retriever and an extractive answerer) with 41 eval questions, 7 of which have no answer in the corpus. Metrics are deterministic and offline: hit@k, MRR, citation correctness, citation precision, whether the answer contains the fact, and abstention accuracy. The gate prints a table with baseline, current, delta and tolerance per metric.
Not done yet
The corpus and questions are mine and small, so the scores say the gate works, not that any real app is good. The baseline abstention accuracy is 0.71, which is low and left as it is. It is not published to PyPI.
Architecture highlights
- Degrading the reference app on purpose fails the gate with a readable diff: the whitespace tokenizer drops citation correctness from 0.94 to 0.50, and top_k=1 drops hit@k from 1.00 to 0.97. Real output is in the repo
- A metric that goes up is not a failure: abstention accuracy rose under the whitespace tokenizer, and the report shows it instead of hiding it
- 146 tests assert exact numbers and cover empty corpus, unicode, ties in ranking, tolerance boundaries, a missing baseline, NaN and infinity, and malformed lines reported with their line number
- An independent audit showed the gate could be passed by citing every passage, by deleting eval questions or by --tolerance inf. Fixed: a citation precision metric, an answer length cap, the eval set hash stored in the baseline and finite-number checks. Exit code 1 now means only a regression
- The LLM-judge metric is only an interface and an example stub, off by default and not used in CI. I did not wire a real vendor