Measuring support quality without vanity metrics

Most support dashboards are full of numbers that go up when things go well, go up when things go badly, and cannot tell you which of the two is happening. They survive because nobody has to defend them: a metric that always looks acceptable never generates an uncomfortable meeting. The cost is that the queue can be deteriorating for a month before any chart notices.

The vanity list, and what each one hides

None of these numbers is fake. Each one measures something real and then gets used to answer a question it cannot answer.

Deflection rate deserves its own mention, because it is the one most often reported as a quality number when it is a capacity number. It counts contacts that did not arrive, and a customer who gave up counts the same as a customer who was helped — the difference is worked through in deflection rate vs resolution rate.

The replacement set

1. 01Resolution rate on a strict definition. Conversations resolved divided by conversations attempted, where resolved means no human agent replied, conversations the customer never engaged with are excluded rather than counted as successes, and a resolution is withdrawn if the conversation reopens.
2. 02Reopen rate inside a stated window. The window is part of the metric: a reopen rate without a number of days attached is not comparable to anything, including its own value last quarter.
3. 03Contact rate, per order or per active account. This is the only number in the set that can fall for a good reason. Volume rises with the business; contact rate rising means the product or the content got worse.
4. 04Backlog age as a distribution. An average age of nine hours is compatible with eleven conversations answered instantly and one waiting four days, and it is the four-day one that writes the review.
5. 05Cost per resolved contact, with the reason for the movement attached.

Response times, measured properly

Three corrections turn first-response time from decoration into a signal. Report the median and the 90th percentile rather than the mean, because the mean is set by a handful of conversations that sat over a weekend. Measure business hours only, or you are grading yourself on the clock rather than on the service. And measure it on conversations that reached a human, because mixing in instant AI replies produces an average that is technically true and operationally meaningless.

The 90th percentile is the one to watch week over week. A median that holds steady while the 90th percentile climbs is the classic signature of a queue where the routine work is fine and the hard work is quietly rotting.

The only version of CSAT worth reporting

A single satisfaction score across a mixed queue tells you almost nothing you can act on. Split by how the conversation ended — resolved by AI, or resolved by a person — and the same responses start answering real questions: whether automation is holding quality, and which topics should not be automated at all.

That split has to be built into the collection path rather than reconstructed afterwards. In CustomerEagle a rating is a whole integer from 1 to 5, one per conversation, and it is stored together with whether the conversation was AI-resolved, so the breakdown is available without joining anything by hand. There are two places a customer can rate — the card in the widget and the link in the resolved-conversation email — and both run through the same code path on purpose, so the two channels cannot drift into measuring different things.

One rating per conversation is a deliberate limit: a second submission is rejected rather than averaged in, so a follow-up cannot quietly overwrite the first impression. A low rating also does something beyond reporting. On a conversation that was counted as AI-resolved, a rating of 2 or lower reverses that resolution — it comes out of usage and is credited back if it was already invoiced, which is the same settling-window mechanic described on how we measure AI resolutions. A satisfaction score that can cost the vendor money is a satisfaction score the vendor has an interest in collecting honestly.

• Report the response rate next to the score, every time. A 4.6 from 6% of conversations and a 4.3 from 40% are not comparable, and the second one is the more useful number.
• Report the two segments separately and never publish a blended score without both counts underneath it.
• Watch the direction rather than the level. A drop of a third of a point in the AI-resolved segment is a content problem with a date on it.
• Read the comments. They are the only part of CSAT that tells you why, and they are usually the part nobody has an owner for.
• Disclose the bias when you present it upward: customers who rate are self-selected, and conversations that ended badly are less likely to reach the rating step at all.

Cost per resolved contact, with a consistent denominator

Calculate cost per resolved contact as the support costs included for a period divided by conversations confirmed resolved in that same period. State whether costs include staff time, software, AI usage, review and allocated setup work. Keep the outcome definition consistent and include follow-up effort. Closing more conversations without checking their outcomes can make the reported figure look better while leaving customers with unfinished work.

Read this figure alongside contact rate, repeat contacts and answer quality. A clearer tracking page can reduce incoming questions while fixed staffing costs stay the same, so fewer contacts do not automatically produce a lower average cost per resolution. Compare equivalent periods and distinguish released capacity from actual spending reductions. If part of your cost is billed per resolution, its counting rule also matters; review the definition in what is an AI resolution. For a worked evaluation that includes review and follow-up time, use the 30-day AI support ROI pilot.

The one-page weekly report

Seven numbers, one page, each with a note on what it would look like if it were being gamed. If a number cannot be defended in that last column, it does not belong on the page.

The last row is the one that turns a report into work. Grouped handoff reasons are a content backlog in priority order, and the top reason on Monday is usually an article that can be written on Tuesday.

Instrumentation is the hard part

A metric is only as good as the event behind it, and most reporting arguments are really arguments about events. Before a number goes on the page it needs three things: a defined event with a written trigger, a named owner who is allowed to change the definition, and an export you can check against real transcripts.

• Attribution by message author, not by conversation outcome. Whether a human replied is a fact about the messages; anything derived from a status field will drift.
• A dated definition file. When a number moves, the first question is whether the definition moved, and that question should take thirty seconds to answer.
• Sampling, weekly. Ten conversations read end to end will tell you more about quality than any chart, and they are also how you find out that a metric is measuring the wrong thing.
• An export with transcripts attached, so any number in the report can be traced back to the conversations that produced it.

None of this requires a data team. It requires deciding which seven numbers you will defend, writing down what each one means, and being willing to publish a number that goes the wrong way. Where the reporting for the AI side of this lives in the product is on the features overview, and the definitions our own numbers use are on how we measure AI resolutions.