How we evaluate LLM applications in production
A practical evaluation framework for teams shipping copilots and RAG systems — without inventing a research lab.
Every team shipping an LLM product eventually asks the same question: how do we know it's actually working?
The answer is not a leaderboard score. It's a portfolio of evaluations, run continuously, that correspond to how your users experience the system.
We break evaluations into four layers. Layer one is unit-style: given a specific input, does the model produce the expected structured output? Layer two is retrieval quality: are we surfacing the right documents to ground the answer? Layer three is response quality, scored by a stronger model against a rubric. And layer four is human judgment on a stratified sample.
The most common mistake is over-indexing on the flashy last layer while skipping the boring first one. Structured-output evaluations catch 80% of regressions and cost you a few cents to run.
The second most common mistake is running evaluations only at release time. Production drift is real — user behavior, index freshness, and upstream model updates all shift the ground under you. Run your eval suite on a schedule, and gate deploys on it.
Marcus builds production LLM systems. Formerly staff engineer at two AI-first startups and a hyperscaler.
More from MarcusRelated articles
Guardrails for enterprise AI, without the theater
How to build AI governance that actually reduces risk — and doesn't just generate a policy binder nobody reads.
Data contracts are boring — and that's the point
Why the least glamorous idea in the modern data stack is quietly the highest-leverage one for teams tired of 3AM pages.
FinOps for the modern lakehouse
The five levers that actually move data platform cost, ranked by impact — and the three that don't.