The demo went well. The support assistant handled the refund question, cited the right policy, kept a reasonable tone. Everyone signed off.
Six weeks later it's quietly wrong maybe one time in eight. Not catastrophically wrong — just confidently citing a policy clause that was superseded in March, or hedging on questions it used to answer cleanly. Nobody filed a bug. The dashboards are green. Latency is fine, error rate is fine, uptime is fine.
Then a customer escalates, someone pulls the transcripts, and the room realizes this has been happening for weeks.
This is the characteristic failure mode of AI products, and it is not the one most teams are instrumented for. Traditional software breaks loudly: a null pointer, a 500, a stack trace with a line number. AI degrades quietly. It keeps returning a well-formed, plausible-sounding response the entire time it's getting worse. There is no exception to catch, because nothing threw.
Why AI regression isn't the regression you know
In deterministic software, a regression is a test that used to pass and now fails. You get a red build, a diff, and a culprit commit. The whole discipline of CI exists because that signal is cheap and unambiguous.
AI systems break that model in three ways.
The output is a distribution, not a value. The same input can produce different outputs on different runs. So "did this change break something?" stops being a yes/no question and becomes a statistical one: did the rate of good answers drop, and by enough to matter? You cannot answer that by eyeballing a handful of responses. Ten spot-checks cannot distinguish an 85% success rate from a 70% one with any confidence.
"Correct" is a judgment call. For a lot of what these systems do — summarizing a claim, drafting a reply, deciding whether to escalate — there's no single right string to assert against. Correctness depends on tone, groundedness, policy compliance, and whether the answer is actually useful. Those are real, checkable properties, but they aren't assertEquals.
The foundation moves under you. Your code can be byte-identical week over week and your product can still get worse, because a dependency you don't control changed.
That last one deserves its own section, because it's the part teams consistently underestimate.
Where AI quality drift comes from: four sources
1. The model
Hosted models get updated. Sometimes with a version bump and a changelog, sometimes with a silent weight refresh behind the same endpoint name. The most cited evidence here is still Chen, Zaharia, and Zou's 2023 study, which tracked the "same" GPT-4 service over a few months: on identifying prime versus composite numbers, the March version scored 84% and the June version scored 51%. Same API, same prompt, half the accuracy. Their broader finding was that instruction-following degraded, which cascaded into everything built on top of it. Nothing on the customer's side changed. Nothing on the customer's side could have changed.
2. Your own prompts
Teams edit prompts constantly — a clarifying sentence here, a reordered example there — and treat it as copywriting rather than as a change to program logic. It is a change to program logic, and an unusually brittle one. Sclar et al. (ICLR 2024) held the task and the model fixed and varied only prompt formatting in few-shot settings — separators, casing, spacing. On LLaMA-2-13B they measured spreads of up to 76 accuracy points across formats a human would call equivalent, and found the sensitivity persisted at larger model sizes and after instruction tuning. You do not need to be anywhere near that unlucky for a Tuesday-afternoon prompt tweak to cost you real accuracy on a segment nobody thought to check.
3. Your data
RAG systems inherit the quality of what they retrieve. Someone adds 400 documents to the knowledge base, or an old policy PDF stays indexed after being superseded, or a chunking config changes. The model is doing its job faithfully — it's grounding on the wrong source. This one is especially insidious because the answer looks better sourced, not worse.
4. Your users
Usage shifts. A feature that launched for one workflow gets adopted for a different one. New customer segments arrive with vocabulary, formats, and edge cases your original test cases never contained. The system didn't change; the distribution of what you're asking it to do did.
Any one of these can move quality a few points. They compound, and they compound between releases, which means the gap between "when it broke" and "when you noticed" is measured in weeks.
Why dashboards and spot-checks miss AI regressions
Uptime and latency dashboards measure whether the system responded, not whether it was right. A model can degrade badly while p99 stays flat and the error rate stays at zero. Every request succeeded. That's exactly the problem.
Spot-checks are sample sizes of five to ten, chosen non-randomly, usually by the person who built the thing and knows which prompts work. They're useful for catching total breakage and nearly useless for catching a ten-point drop.
Spreadsheets of test cases are the most common serious attempt, and they fail for operational reasons rather than conceptual ones. The sheet works for one release. Then it's out of date, the results from the last run were overwritten, nobody remembers which model version column D refers to, and the one person who maintained it is on another project. Without a preserved baseline there's no such thing as a regression — only a number with nothing to compare it to.
User complaints are the worst signal of all, because of who generates them. Users who complain are your engaged users. The ones who quietly conclude the feature isn't reliable and stop using it never file anything. By the time complaint volume is legible as a trend, the damage is in your retention numbers, not your bug tracker.
How to catch AI quality regressions before users do
The mechanism isn't complicated. It's just unglamorous, and it has to run every time something changes.
A fixed test set that reflects real usage. Not the happy path. The refund edge case, the ambiguous policy question, the customer who writes in three languages in one message, the input that previously produced something embarrassing. Every incident should end with a test case, permanently.
A consistent judge. Whether that's an LLM grading against a rubric, a deterministic check, or a human review queue for the calls that need domain expertise — what matters is that it's applied the same way every run. A judge that drifts is worse than no judge, because it produces numbers you'll trust.
A stored baseline. Last release's scores, kept, so this release's scores mean something. This is the piece spreadsheets lose first and the piece that turns a number into a decision.
A trigger on every change. Model version, prompt edit, retrieval config, new data source. If it can move quality, it runs the evals. The failure mode of eval programs isn't bad methodology, it's running them only before big launches — which is precisely when drift has had the longest to accumulate unobserved.
Failure reasons, not just scores. "78%" tells you nothing actionable. "Failures cluster in multi-turn refund conversations where the policy was updated after the index was last built" tells you what to fix. The point of an eval isn't the number, it's the diagnosis.
Why AI evals can't stay locked in code
Here's what makes this harder than it should be: most eval tooling lives in code, which means it lives with engineering, which means the people who best understand what "correct" means for your product often can't touch it.
Your claims specialist knows which edge cases are genuinely dangerous. Your policy lead knows which phrasing creates regulatory exposure. Your PM knows which failures customers will forgive and which ones lose the account. Ask an engineer to encode all of that into a rubric and you've introduced a translation layer between the people who hold the quality bar and the system enforcing it — and translation layers lose information.
The teams that get this right don't have better prompts. They have a shorter path between "the domain expert noticed something wrong" and "there's now a test for it that runs on every release."
That's what an AI quality loop is: define what good looks like, run evals on every change, diagnose the failures with reasons attached, decide with evidence, repeat. Not because it's rigorous for its own sake, but because it converts "should we ship this?" from a room full of instincts into a question with an answer.
The alternative isn't that you never find the regression. It's that your users find it first.
TryEval is a no-code AI quality loop for product teams — evaluate AI systems, review failures, track regressions, and make confident ship/no-ship decisions without writing code. Start evaluating or request a demo.
Sources
- Chen, L., Zaharia, M., & Zou, J. (2023). How is ChatGPT's behavior changing over time? arXiv:2307.09009
- Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. ICLR 2024. arXiv:2310.11324