← Back to all work

AI Case Quality Auditor

Making AI scores useful for coaching

Support managers needed to review more closed technical cases without losing the coaching value of each evaluation. I shaped how scores, evidence and human review fit together.

Role
Senior Product Manager
Partners
Customer QA leads, Engineering, QA
Scope
Scoring criteria, calibration, production rollout

The problem

Support managers could only review a small sample of closed technical cases by hand. They wanted to cover far more cases, but a bare number would not help them coach an engineer.

A score is only useful in a one-on-one if the manager can see why it was given and point to the moment in the case that caused it.

My role

I owned the product as Senior Product Manager. I worked with customer QA leads on what good looks like, with engineering on scoring and evidence extraction, and with QA on testing before and during production rollout.

Decisions I owned

The decision that shaped the product

Every score comes with a reason and the evidence behind it

I decided that the AI would never return a score on its own. Each criterion carries a written rationale and a citation to the exact part of the case it relied on. An authorized reviewer can revise the score when the case context warrants it, because some context lives outside the case record.

  • Rationale for every criterion
  • Cited case evidence
  • Score revision by authorized reviewers

Criteria written as observable behavior

With customer QA leads I rewrote rubric items so the evidence for each one could be found in the case. "Was empathetic" became checks a reviewer and the model could both point to.

Calibrate before rollout

We compared AI evaluations against human reviews of the same cases before turning scoring on in production, and used the gaps to refine criteria and prompts.

Human revision is part of the product

Revision is limited to authorized reviewers and keeps the original finding visible, so agents see both the AI's reasoning and the manager's.

From AI finding to coaching

  1. AI finding

    The auditor scores a closed case against each criterion and writes what it found.

  2. Evidence review

    The manager opens the cited note or message and checks it against what they know about the case.

  3. Coaching conversation

    The manager keeps or revises the score and brings one specific, evidenced point into the engineer's one-on-one.

Illustrative example
Case 40817 · SSO sign-in fails after certificate rotation · Agent: R. Mehta (fictional)

Click Finding, Evidence and Manager Review to follow one review

Resolution verified4 / 10

AI finding

The agent changed the identity provider certificate setting and closed the case. The case record has no successful sign-in retry and no confirmation from the customer that the error stopped.

Why it matters

Without a verified fix, the customer may reopen the case or escalate, and the closure counts toward resolution metrics it may not have earned.

What's happening

The AI scored this closed case 4/10 on one criterion and explains why in plain language.

What customers told us

30+ minto audit a multi-agent case by hand
10–15 minwith AI Case Quality Auditor
50–67%less time per audit

Customer-reported. The reviewer still checks the AI's findings, so the time saved is in finding and citing evidence, not in skipping the review.

What calibration taught us

Calibration showed that the AI and human reviewers mostly differed where context sat outside the case record, such as a call, a chat thread or a linked ticket. That is why reviewer revision exists.

It also showed where rubric wording left room for interpretation. Tightening those criteria improved agreement more than changing the model did.

Evals and governance

AI can sound sure of itself and still be wrong. So for every AI feature I ship, I answer three questions: is it right, can people check it, and do humans stay in control?

  1. Evals

    Question 1

    Is the AI right?

    Before anyone relies on a score or a reply, I measure how often it matches what an expert would say.

    Used inAI Case Quality AuditorAI Agent Partner

  2. Governance

    Question 2

    Can people check it?

    A score nobody can verify won't be trusted, so the AI has to show its work.

    Used inAI Case Quality AuditorAI Agent Partner

  3. Governance

    Question 3

    Do humans stay in control?

    The AI informs the decision. A person makes it.

    Used inAI Case Quality AuditorAI Active Case Agent

See how I test and govern AI across products →

See the workflow in practice