What is LLM evaluation in CI?

LLM evaluation in CI runs a versioned golden set of real questions on every change that can alter the application’s answers, and blocks the build when an exact check breaks or a judged score falls beyond measured noise. A failed run names the cases; a passing one shows known cases still work, not that the system is correct.

Harmless-looking changes alter answers: a reworded prompt, a chunk size, a tool manifest edit, a model swap that the AI gateway releases only when the golden set passes. This is layer 4 of the enterprise AI platform architecture; DeepEval and Langfuse details date from October 2026.

The evaluation gate at a glance

What to decide The rule Where it lives How to test it
The golden set Real questions, expected documents, one behaviour each A reviewed JSON file beside the code Count cases per behaviour: refusals, cross-tenant questions
Metrics per layer Judged retrieval, grounding, relevance on answer cases; exact checks for the rest Plain tests; DeepEval via evaluate() Remove an expected document: its case fails retrieval
Baseline and threshold No worse than a reviewed baseline, per metric A reviewed baseline file Degrade the prompt on a branch: exit 1, cases named
Noise and repeats Tolerance from eight or more unchanged runs; none for exact checks The baseline file Rerun main several times: it should rarely fail
The judge Model, version, eval mode, template pinned; calibrated on human labels Metric configuration and baseline Change the judge model: exit 2
Trust in a run Changed set or judge, missing score, error: never a pass The gate’s first step Fail one judge call: exit 2, not 1
Production traces A bad answer becomes a case, with trace and prompt version Langfuse traces; the case’s origin From a case, open its trace and prompt version

What a golden set case holds

A golden set case is a question someone really asked, or a domain expert would, plus what must happen: the documents that should support the answer and one behaviour, answer, refusal or escalation. Two illustrative cases, in the field names of DeepEval’s goldens:

[
  {
    "input": "How long do customers in Spain have to return an order?",
    "expected_output": "Thirty days from delivery.",
    "additional_metadata": {
      "id": "returns-window-es",
      "behaviour": "answer",
      "tenant": "tenant-a",
      "expected_sources": ["policies/returns-eu"],
      "origin": "trace 3f9c2a, prompt returns-assistant v12"
    }
  },
  {
    "input": "What penalties does the Northwind contract set?",
    "additional_metadata": {
      "id": "other-tenant-contract",
      "behaviour": "refusal",
      "tenant": "tenant-a",
      "expected_sources": [],
      "origin": "security review"
    }
  }
]

Expected sources name documents, not chunks, so re-chunking leaves cases valid. expected_output is the reference answer DeepEval’s contextual metrics need, so judged metrics run on answer cases only; the others get exact checks. A set worth gating on holds four kinds: answers the documents support; refusals, where no document answers; cross-tenant questions, refused too; escalations, such as an erasure request that must reach a person. Add hard and hostile ones: ambiguous questions, near-duplicate documents, personal-data extraction, instructions planted in a document.

The file changes by review, like code, with one owner. Cost: ongoing curation. Test: parse it in CI; fail on a duplicate ID, an answer case without sources or a case without an origin.

How many cases a golden set needs

A few dozen reviewed cases, refusals included, can gate known behaviour: each is a named check that a change can break. They cannot measure a rate. With forty cases and thirty-six passes, the Wilson score interval runs at the usual ninety-five per cent level from a little over three in four to about twenty-four in twenty-five, and assumes a random sample, which a curated set is not.

So the gate can say “these cases still pass”, never “the assistant is right nine times in ten”. A small change in the pass count is invisible at this size, and so is a decline on questions the set does not hold. Grow the set from production failures: every bad answer that reached a user is a case it lacked. Test: for each recent production failure, find its case.

Metrics per layer, and where DeepEval fits

Measure each layer separately, so a failure points to its cause.

  • Retrieval: are the expected documents among those retrieved? Compute it in code from expected_sources. DeepEval’s contextual recall and contextual precision judge retrieval_context against expected_output: recall, whether the reference answer’s statements can be attributed to the retrieved passages; precision, whether relevant passages rank above irrelevant ones. Under a tenant filter, use the RAG guide’s recall test against exact search.
  • Grounding: faithfulness has a judge check the answer’s claims against the whole retrieval_context: a claim counts as truthful unless it contradicts the passages, and ambiguous claims count unless penalize_ambiguous_claims is set. In my reading, unsupported but compatible claims pass, so the citation check stays in code.
  • Relevance: answer relevancy, from input and actual_output: the share of the answer’s statements relevant to the question.
  • Behaviour: refusal, escalation, no other tenant’s document cited, personal data masked. Exact checks, never a judge.
  • Budgets: latency and cost per request, each a threshold checked like an exact check.

These metrics are LLM-as-a-judge by default, passing a case at 0.5. DeepEval advises gating with unit tests, where assert_test fails any case under its threshold, yet admits cases near a threshold flip between runs. For judged scores I would depart from that: collect them with evaluate(), save each run with results_folder and let a gate script decide. The cost: owning that script and its repeat loop, without DeepEval’s baseline comparison, which runs on its hosted Confident AI service.

The gate: a reviewed baseline and a measured tolerance

I would keep the baseline in a repository file: the set’s hash, the judge’s identity, the application’s versions and, per metric, the mean and a tolerance. Start it from today’s scores, so the first gate is “no worse than today”, and review it; never write it from the run being tested, as Langfuse’s CI documentation also warns.

Exact checks I would gate per case: a refusal case that answers, in any repeat, blocks. Judged metrics I would gate on the mean, though both tools document per-case gates: with three independent repeats, a case that passes half its runs fails all of them one gate run in eight. The mean can hide one broken judged case, so anything that must never break gets an exact check, and the gate prints the judged cases that fell most.

For the tolerance, I would run the unchanged version eight times or more, with independent repeats, none served from DeepEval’s cache: the baseline is the mean of those runs, the tolerance the largest gap between their means. An unchanged version still fails now and then: with roughly normal, independent noise, four judged metrics and eight baseline runs, about one run in eighteen; with only five, one in six. Averaging independent repeats divides a mean’s spread by the square root of their number (NIST).

I would run the exact checks and a judged subset on every pull request, the full set with repeats before release, and nightly, for what skips pull requests: documents in the index, a prompt label moved in Langfuse, a model’s lifecycle state on Bedrock. Before the third product ships, the build order leaves out a large judged suite: a few dozen reviewed cases with exact checks can gate.

Worse, or cannot compare: what a failed run means

The fixes differ, so a failed gate must say what failed. pytest, which DeepEval’s test runner plugs into, separates failing tests (exit code 1) from internal errors (3); I would give the gate three outcomes, exit code 2 being my own convention:

  1. Cannot compare, exit code 2: the set or the judge differs from the baseline, so re-baseline; or a judge call failed or a score is missing, so rerun. The gate must catch errors to return 2: an uncaught exception exits with 1.
  2. Worse, exit code 1: an exact check or a budget failed, a judged mean fell beyond its tolerance, or the application errored on a case, with the cases named.
  3. Pass, exit code 0, still listing the judged cases that fell most.

I would have a pull request that edits the set carry its new baseline, built from the base branch and reviewed with the cases. Leave DeepEval’s ignore-errors option off: a gate that skips errored cases has stopped measuring. Test: fail one judge call; the job exits 2.

LLM judges: pin them, calibrate them, keep exact checks exact

A judge is part of the measurement, so pin it like a dependency: model, DeepEval version, eval mode and any template, recorded in the baseline. An unset model means whatever the CI machine has configured, or OpenAI’s gpt-5.4, called outside the gateway and its EU-only rule. I would pass a custom model that calls the gateway and keep the llm eval mode: hybrid and system_one call TypeSafe AI’s Jev, and hybrid falls back to the LLM when a Jev call fails.

Zheng et al. (2023) found position and verbosity biases and judges misled by the answers they graded, and note that absolute scores are likely to fluctuate more than pairwise comparisons when the judge model changes. Hence a re-baseline for every judge change, and deterministic checks for anything exact.

Calibrate before trusting: have experts label a sample, compare it with the judge and read false passes first; Langfuse’s score analytics reports agreement between two scores on the same traces. Test: give the judge a case a person failed; it fails it.

Production traces become golden set cases

Tie each production trace to what produced it: linking prompts to traces in Langfuse gives metrics per prompt version, and a release set to the commit names the code. When a trace shows a bad answer, reproduce it, fix the cause and add the case, with the trace ID and prompt version in origin. Langfuse can create a dataset item from a trace; I would use it as an inbox for cases to review, which erasure must then reach too.

A production question can hold personal data, and a copied case puts it in Git, where erasure is hard: rewrite it, and keep only the trace ID. Langfuse also serves the version labelled production when no label is requested (version control), so moving that label deploys a prompt without a pull request: run the set before moving it.

In practice

On the ENSO platform I specified evaluation gates in CI built on DeepEval, golden sets and Langfuse tracing, as one layer of a platform that also includes a gateway, retrieval and governed agent actions. The wider architecture is described in the enterprise AI platform reference guide and in the ENSO case study, and evaluation gates of this kind are part of the enterprise LLM platform service.

Trade-offs and limits

Evaluation costs judge calls per metric, case and repeat, and each re-baseline costs eight or more full runs. A baseline goes stale when the set or the judge changes, and old scores no longer compare.

Golden sets go stale as products and documents change, and a team can overfit by tuning prompts until its own set passes. Refresh the set from production faster than you tune against it.

A product with no traffic yet has no traces: start from experts’ questions and from what the documents do not answer, and replace them as real questions arrive.

A passing gate shows that known cases still work, not that the system is correct, so human review of real conversations stays.

Common questions

What is a golden set for an LLM application?

A versioned file of real questions kept beside the code, each with the documents that should support the answer and the expected behaviour: answer, refusal or escalation. It runs on every change to a prompt, model, retrieval setting or tool manifest, and changes by review.

How many test cases does a golden set need?

A few dozen reviewed cases, refusals included, can gate known behaviour: each is a named check. They cannot measure a rate: with forty cases and a pass rate near nine in ten, the plausible range is about a fifth wide. Grow the set from production failures, not towards a number.

How do you handle flaky LLM evaluation results?

Never in exact checks: a refusal case that sometimes answers is a defect. For judged metrics, I would gate on each metric’s mean against a tolerance from eight or more independent runs of the unchanged version, and accept a rare false red. Repeats narrow the noise but add no coverage.

Can you trust an LLM judge?

Within limits. Pin its model, library version, eval mode and template, and re-baseline when any changes. Calibrate it against human labels, reading false passes first. Research has found position and verbosity biases and judges misled by the answers they grade, so anything checkable exactly stays deterministic.

Sources