Testing evidence completeness in team roster without a convenience sample

Research findings

## Question and scope The study measures the record and the handoff, not the professional judgment of operations manager or the legal outcome of the work. State how duplicate requests are identified and whether they are merged, linked, or counted separately. If a definition changes during the review, keep the earlier classification and apply the new rule only to later cases. If the study is exploratory, label it exploratory and do not present the result as a stable baseline. Link the observation to the exact field in the task queue and to the task record that supports it. State the observation window, the sites or teams included, and the exclusions before collecting the first case. Record the opening count, additions, withdrawals, resolutions, and unresolved items at the end of the window. If a definition changes during the review, keep the earlier classification and apply the new rule only to later cases. Name the population, the observation window, and the systems included in the extraction. Do not let a summary written later replace the evidence that was available at the first decision.

## Population and unit The unit is one team roster case, from the opening request to the verified close, and linked events stay with that case. A case that moves between owners stays one case unless the process defines a new requirement at the handoff. A correction should be classified by the original case and by the correction event, with both dates retained. Separate cases opened before the window from cases opened and closed inside it, and report both counts. If the case moved between owners, include both owner periods in the review note and name the handoff time. The unit is one team roster case, from the opening request to the verified close, and linked events stay with that case. The population should include unresolved cases, not only closures, or the study will overstate completion. Define how a case that waits for client sponsor is counted: paused, open, or excluded, but not silently dropped. A case that never received an owner still belongs in the population; do not drop it because it is difficult to classify. Keep the review note with the case so the next sampler does not have to reconstruct the decision from memory.

## Definitions and measures A pause should have a reason, an owner, and a review date; otherwise it is an unowned case with a different label. Record the source system, the record identifier, the extraction time, and the reviewer for every extracted value. Record the person who approved the definition and the date it became effective. Define whether the measure counts requests, cases, or destination updates, and use that unit consistently. When a value is missing, record the missing state and the reason instead of leaving an empty field in the extract. Define how a correction after close is linked to the original case and whether it creates a new case. Test the data dictionary on a small set of cases and revise ambiguous fields before the main sample. Define whether a case that waits for a third party is open, paused, or excluded, and apply that rule to every owner. Define whether the measure counts requests, cases, or destination updates, and use that unit consistently. At Outsourced Company, the team roster review should end with a dated note and a named owner for the next check.

## Data dictionary and evidence Test the dictionary on five cases before the full sample; ambiguous fields should be fixed before collection, not after. Record the extraction time separately from the event time so a late entry is visible. Record the data extraction date and the version of the access management console used for the population count. Use a separate field for reviewer notes so the coded values stay comparable. If the evidence is sensitive, record the approved reference and access path rather than copying the content. The data dictionary should name the allowed values for status, exception reason, owner, and close evidence. Use a controlled list for exception reasons; do not let free text become the analysis category. Keep the original evidence reference next to the coded value so the reviewer can verify the classification. Use a separate field for reviewer notes so the coded values stay comparable. The note should distinguish what was observed, what was inferred, and what still needs an owner decision. Mark missing, unavailable, and confirmed negative values separately; collapsing them hides the difference between no evidence and evidence of absence. Give every case a study key that is not the client name, member number, or candidate email. Run a five-case pilot before the full sample and revise any field that two reviewers interpret differently. Use a separate field for reviewer notes so the coded values stay comparable. Record this decision in the study log with the case key, the review date, and the person who approved the classification.

Discussion

## Sampling and collection If the population is small, report every case and the reason the sample is exhaustive rather than presenting a false precision. Record the time of each observation so a later reviewer can distinguish a late entry from a changed decision. Do not expand the sample only because the early result looks weak or strong. Record the date of the sample and the version of the queue export. Keep the evidence reference beside the coded value so a reviewer can test the same case without rebuilding the queue. A convenience sample of recently closed cases will overrepresent easy work and underrepresent the tail that needs attention. Collect the exception reason and the action taken before the close so the analysis can separate detection from resolution. If a stratum has fewer cases than planned, report the smaller count instead of replacing it with a convenience case. Keep a table of planned, eligible, selected, and reviewed counts for each stratum. If the source changes after the observation, preserve the earlier value and add a dated note instead of replacing it. A convenience sample of recently closed cases will overrepresent easy work and underrepresent the tail that needs attention. Record the time of each observation so a later reviewer can distinguish a late entry from a changed decision. Do not expand the sample only because the early result looks weak or strong. Use a random start and a fixed interval when the frame is ordered, and record both. At Outsourced Company, the reviewer should be able to explain this step to offshore worker from the case record alone.

Implications and limits

## Analysis and reporting If the sample is small, report the instability and the cases that drive the result instead of a precise-looking percentage. Use case-level review to explain an outlier before treating it as a process signal. Report the number of cases that changed status after the first extraction and how the change was handled. Report the number of cases that drive a small-sample result and the direction of the effect. Use the same note in the case file and the study extract; do not create a second version in a personal document. Keep case-level detail available for review, because a summary chart without records cannot support a decision. A small sample should be reported with the number of cases that drive the result. Write the finding in plain language and separate what was observed from what the team believes caused it. When a rate changes, check whether the case mix or the evidence standard changed at the same time. When a case needs an exception, record the reason, the owner, and the date before continuing the review.

## Limitations The cited public sources frame recordkeeping, privacy, security, or control practice; they do not provide a local result. The record may be incomplete or delayed, so the method cannot infer work that never entered the queue. Name the follow-up evidence that would reduce the largest uncertainty. State what a follow-up study would measure and which population it would include. Keep the measurement note short enough to read with the case and specific enough to support the reported quality review pass rate. The method also cannot observe work that never entered the queue, which may be the most important missing population. Changes in staffing, access, or the team roster during the window can create a difference that is not a process effect. If the reviewer also performed the work, note the independence limitation and use a second reviewer for a subsample. State what a follow-up study would measure and which population it would include. If two reviewers disagree, preserve both classifications until the definition or the evidence is resolved.

## Review and next action If the action affects offboarding gap, route it to offshore worker and record the approval before implementation. If the result is inconclusive, say so and choose a narrower question rather than claiming a trend. If the action requires a policy or access change, route it to the owner and record the decision. Link the action to the case evidence that prompted it, not only to the meeting notes. Link the observation to the exact field in the team roster and to the quality review that supports it. A quality review should end with an owner, a date, and one testable change, not a list of intentions. Repeat the protocol with the same definitions and window, then compare the same measures. Record the owner, the due date, and the exact change that will be tested at the next review. Schedule the next review before the action begins, not after the result arrives. Do not let a summary written later replace the evidence that was available at the first decision.