Beyond CI/CD: Continuous Calibration for LLMs in Production

CI/CD was designed for deterministic systems. You write code, the code either does what you intended or it doesn't, you run tests, you deploy. The test suite is a proxy for correctness because the underlying system is predictable: same inputs, same outputs, always.

LLMs break this assumption in two places simultaneously. Most teams discover this three months into a production deployment — when a system that worked great in testing has quietly degraded and nobody noticed because the traditional monitoring stack has no way to see it.

The Dual Uncertainty Problem

User uncertainty is the one teams anticipate. Natural language is inherently unpredictable — you can't enumerate every way a user might phrase a request, and your test suite will not cover the real query distribution. This is solvable with good evaluation design, but it requires deliberate effort.

Model uncertainty is the one teams underestimate. LLM APIs are probabilistic. Identical inputs can produce different outputs across invocations, API versions, and the silent model updates providers ship without notice. The model you evaluated against in January may not be the model you're calling in March — even if the endpoint name hasn't changed.

Together, these create what I call Trust Debt: the compounding gap between the behavior you validated at deploy and the behavior you're actually running in production. You shipped with confidence. The debt accumulates silently.

The CC/CD Framework

The framing I've found useful: Continuous Calibration / Continuous Development. These are parallel tracks that run concurrently, not sequentially.

Continuous Development is what you know: code changes, prompt changes, feature releases. You ship and verify.

Continuous Calibration is what most teams skip: ongoing behavioral monitoring that verifies the system is still doing what it was designed to do — independent of what you changed. A system with zero code changes can still drift because the model underneath it drifted.

CC runs on a fixed evaluation cadence (not triggered by deploys). It compares current behavior against a behavioral baseline. When drift exceeds a defined threshold, it triggers review — not deployment.

The key distinction: CD catches regressions you introduced. CC catches regressions that arrived without your involvement.

The V1–V3 Agency-Control Lifecycle

Production LLM deployments tend to evolve through three stages, and the calibration requirements are different at each:

V1: Copilot. The model suggests; a human decides and acts. Calibration is primarily about output quality — is the suggestion useful? The blast radius of a bad output is low; a human catches it before anything happens.

V2: Assistant. The model acts within a constrained scope; a human reviews after. Calibration expands to include behavioral consistency — is the model making decisions in the same way it was when we approved the scope? The blast radius is higher because actions happen before review.

V3: Agent. The model plans and executes multi-step tasks with minimal human intervention. Calibration must now include: goal alignment (is it pursuing what we intended?), scope adherence (is it staying within its sanctioned boundaries?), and failure-mode monitoring (when it gets stuck, what does it do?). The blast radius here is organizational.

Most teams design their V3 deployment like a V1 deployment with a bigger eval set. That's the wrong mental model. V3 requires calibration infrastructure — continuous behavioral monitoring, sampling pipelines, and defined escalation paths — that simply wasn't necessary in V1.

Statistical Evals: Two Approaches

The implementation detail that most frameworks gloss over: how do you actually measure behavioral consistency?

Rubric Strategy. Define a structured rubric with explicit criteria (relevance, completeness, factual grounding, format adherence). Use a judge LLM to score outputs against the rubric on a fixed eval set at regular intervals. The output is a score vector per sample. You're looking for distribution shift, not individual sample scores — a healthy system shows stable score distributions over time, not identical outputs.

Semantic Similarity Measurement. Embed current outputs and a reference output corpus. Measure cosine similarity distributions. A system that's drifting will show widening similarity distributions — outputs that are more varied, less anchored to the reference behavior. This is particularly useful for open-ended generation tasks where rubric scoring is expensive.

Neither approach gives you a binary pass/fail. Both give you a signal you can threshold. The threshold is a business decision: how much behavioral drift is acceptable before you require human review?

What This Looks Like in Practice

A minimal Continuous Calibration setup has:

1. A fixed behavioral baseline: 200–500 examples spanning the use case distribution and known edge cases, with reference outputs captured at last-approved deploy
2. A scheduled calibration job (weekly minimum, daily for V3 systems) that scores current model behavior against the baseline
3. A distribution comparison that flags when score distributions shift beyond threshold
4. A defined review process when the flag triggers — with a human accountable for disposition
5. A feedback loop that routes reviewer annotations back into the baseline when the new behavior represents an intended change rather than drift

None of this is glamorous infrastructure work. But it's the difference between a system you can defend and a system you're hoping is still working the way it was when you deployed it.

For regulated industries — finance, legal, healthcare — the second framing isn't optional. You don't just need to demonstrate that your AI system worked at deployment. You need to demonstrate it works today, with a paper trail showing continuous verification in between.

That's what Continuous Calibration is for.

For a practitioner-level breakdown of the V1→V3 agency lifecycle, statistical rubrics, and cosine similarity delta measurement, see the engineering deep-dive: Beyond CI/CD: A Technical Guide to Continuous Calibration for LLMs in Production.