Philippines staffing research · Published · Updated
How audit sampling error shapes an outsourced quality review program
A quality score is an estimate from a sample, not a measurement of every interaction. Programs that ignore sampling error, reviewer disagreement, and exclusion rules make confident decisions from thin evidence.

Research findings
A quality program rarely reviews every ticket, call, or transaction. It reviews a sample and reports a score as if the sample were the whole population. That shortcut is useful, but it carries error, and the error grows when the sample is small, convenient, or poorly defined. The first step toward a trustworthy program is to admit that the score is an estimate. That estimate should be labeled as one in every report the desk sends.
The sampling frame is the list from which the sample is drawn, and it decides what the score can claim. If the frame excludes escalations, reopened tickets, or work completed on a weekend, the score describes the remaining work only. Write the exclusions down and report them with the score. A narrow frame presented as the whole operation is a misleading result even when the math is correct. Report the excluded count beside the score so the gap stays visible.
Random selection protects against a reviewer's unconscious preference for tidy work. A convenience sample drawn from the first ten tickets of the day will overrepresent simple cases and morning staff. Use a random number generator or the system's own sampling tool, and keep the seed or list so the draw can be repeated. Repeatability is what makes the sample auditable. Keep a dated copy of the draw with the review file.
A larger sample narrows uncertainty, but it never removes it. The right size depends on how close the score needs to be and how much variation exists between teams. A process with wide variation needs more items to produce a stable estimate than a process that is already consistent. State the uncertainty beside the score instead of presenting a single number as fact. A confidence interval is more useful than a decimal place.
Stratify the sample by team, queue, shift, and work type when those groups can differ. A single random draw can miss a small but risky queue entirely, especially when that queue handles a fraction of the volume. Allocate part of the sample to each stratum and report the results separately. Aggregation can hide a failing group behind a strong average. Review the stratum results before discussing the combined score.
A defect must be defined before the sample is scored. If reviewers disagree about whether a missing note is a defect, the score measures their preferences rather than the work. Write the definition with examples and a decision for borderline cases. Test the definition on a few items and revise it before the sample begins. A written definition also makes the reviewer's decision easier to challenge.
Discussion
Reviewer calibration is the next source of error. Two reviewers can watch the same call and reach different conclusions about tone, verification, or resolution. Run a calibration set where every reviewer scores the same items and compare the results. Discuss the disagreements and update the rubric. A program that never calibrates is measuring reviewer drift as much as it measures the quality of the work itself.
Decide whether the sampling unit is a ticket, a customer, a call, or a day. A customer with five tickets may appear five times in a ticket sample, which gives that customer more influence than a customer with one. Cluster sampling by day or team can be efficient, but it reduces the effective sample size. Match the unit to the decision the score will support. Keep the unit stable across periods so the trend means something.
A perfect score should prompt a check on the sample, not a celebration. Small samples often produce perfect results by chance, and a narrow frame can exclude the work most likely to fail. Look at the number of items, the exclusions, and the reviewer agreement before trusting the result. A score without that context is a headline, not evidence. Report the count of items reviewed before the percentage.
Implications and limits
Report the sample size, the frame, the exclusions, and the interval around the score. If the program cannot calculate an interval, report the count of reviewed items and the count of defects so the reader can judge. Avoid ranking teams on differences that are smaller than the sampling error. A ranking that flips each month is a sign the sample is too thin. State the interval in the same sentence as the score.
A finding should lead to a named corrective action, an owner, and a due date. Sampling error does not excuse inaction on a confirmed defect, but it should shape how broadly the fix is applied. Fix the specific case first, then decide whether the pattern justifies a process change. Record the decision so the next sample can test whether it worked. Review the action at the next sample instead of waiting for the quarter.
No sampling program can prove that every interaction was correct. It can estimate the defect rate, detect patterns, and test whether a change helped. The program should state those limits in its own report rather than implying a guarantee. A modest claim that matches the method is more credible than a precise score with no visible basis. Write that limit in the report's method note, not in a footnote.
Build a sampling frame you can defend
Start with the system export that holds the work, not a list prepared by the team being reviewed. Confirm that the export covers the full period and that the record count matches the system's own total. Write down the filters used, the date the export was taken, and the person who produced it. A frame that cannot be reproduced makes every later comparison uncertain. Confirm that the export includes reopened work and late closures.
List the exclusions and the reason for each one. Test items, duplicate records, and work still in progress may be reasonable exclusions, but they change what the score describes. Report the excluded volume next to the result so the reader can see the gap. If the exclusions are large, the score may describe a minority of the work. If the count cannot be reconciled, do not score the sample yet.
Calibrate reviewers before comparing scores
Choose a calibration set that covers typical work and at least one difficult case. Have each reviewer score the same items independently, then compare the results item by item. Record where the scores differ and ask each reviewer to explain the reason. The discussion should end with a rubric change or a shared interpretation, not a compromise number. Record the rubric version with every calibration result.
Repeat the calibration exercise when the rubric changes or a new reviewer joins. Keep the old calibration set so the team can see whether the change improved agreement. Track the spread between reviewers alongside the defect rate, because a stable score can hide widening disagreement. A program that compares teams without comparing reviewers is comparing noise. Keep the calibration set small enough to repeat each quarter.
Read a perfect score with caution
A perfect score from a small sample is weak evidence, especially when the sample came from one team or one week. Ask how many items were reviewed and how many were excluded before discussing the result. If the count is low, the honest description is that no defects were found in the items reviewed. That statement is narrower than saying the process is defect free.
Look for signs that bad news is being filtered before the sample. A queue with no escalations, a frame with no weekend work, and a rubric with no serious defect category can each produce a clean result. Check the exclusions and the defect definitions before accepting the score. A program that cannot fail is not measuring much. Ask the reviewer to name the evidence behind the clean result.
Connect findings to corrective action
Write the finding in plain language and attach the evidence the reviewer used. Name the owner who will correct the case and the date the correction is due. If the finding suggests a broader process problem, separate that decision from the case-level fix. A broad change based on one item is as risky as no change based on a pattern. Attach the corrected record or a link to it in the finding.
Verify the fix in the next sample and record whether the defect recurred. Close the finding only when the evidence shows the correction happened, not when someone says it will. Report the closure rate and the age of open findings to the client. A corrective action that is never tested is a promise, not a control. Report open findings by age so old promises stay visible.
Questions
How large should a quality sample be?
Large enough to support the decision the score will drive, with the uncertainty stated. A weekly team huddle can use a smaller sample than a contractual service-level report.
Should the desk review the same number of items every week?
Not necessarily. Volume, risk, and the size of the change being tested should shape the sample. Keep the method consistent and explain any change in size.
What if reviewers cannot agree on a defect?
Treat the disagreement as a rubric problem. Add examples, define the borderline case, and recalibrate before using the scores to compare teams.
Can a perfect score be reported?
Yes, with the sample size, frame, and exclusions beside it. Say that no defects were found in the items reviewed rather than claiming the process is perfect.