Proof-of-concept demos create false confidence. Models change, indexes refresh, prompts shift, and tool schemas evolve — yet many teams still treat evaluation as a one-off notebook run. The gap is not a missing dashboard. It is missing ownership of goldens, calibrated judges, and CI gates that fail closed.
What “release gate” means in practice An offline suite runs on every meaningful change: retrieval config, prompt template, model version, or agent tool list. Goldens encode domain expectations — with rubrics where exact match is impossible. LLM-as-judge scorers are versioned dependencies, not magic. When a score crosses a agreed threshold, the pipeline stops. That is qualitatively different from logging a metric for later.
RAG and agents need different regression packs Retrieval programmes need faithfulness, citation, and context precision checks — not only end-to-end answer similarity. Agent programmes need trajectory tests: did the right tool fire, with the right arguments, in the right order? Flakes here are production incidents waiting for peak traffic.
Online monitoring closes the loop Production traces surface edge cases goldens missed. The eval engineer’s job is to promote those failures into the offline suite — not to stare at a flat chart. Observability without dataset refresh is decoration.
What industry surveys report (sources linked on this page) Third-party research consistently shows agents and observability ahead of eval discipline — LangChain (agent engineering survey), Gartner via Sayari (eval tool adoption with separate org vs engineering-team denominators), and Stanford AI Index (documented incidents). Halcer quotes these sources; we do not publish our own forecasts or client outcome metrics.
Where staffing fits Platforms store runs and traces; they do not maintain your domain goldens or argue with a delivery lead about whether 0.82 is good enough to ship. That judgment layer is why eval engineer, AI QA (LLM apps), and red teamer titles are crystallizing — distinct from generic QA or MLOps alone.
Halcer staffs that layer: harness builders and regression pods alongside builders when needed. We do not sell eval SaaS, issue audit opinions, or promise zero hallucinations.