Checking the Validity of an Offshore Work Sample

Research findings

## Research question Does a hiring work sample measure the capability the role actually needs, and does it do so consistently across candidates? The unit is one sample task, one rubric, and one group of candidates for a defined role. The method is a validity review, not a claim that a particular test is scientifically proven.

## Define the construct Start by naming what the sample is supposed to measure. Accuracy? Source verification? Customer writing? Prioritization? Boundary awareness? A single task often measures several things at once, which makes the score hard to interpret. Write the primary construct and no more than two secondary ones. If the role requires both speed and accuracy, decide how the rubric handles a fast but wrong answer. A sample that rewards the wrong construct will select the wrong behavior.

## Check task realism Compare the sample with the actual work. Does it use the same tools, the same kind of source, and the same ambiguity? A clean spreadsheet exercise may not predict performance in a messy inbox. A customer email with no prior context may not reflect the queue. Ask two current team members to complete the sample. If they disagree about what good looks like, the task is ambiguous. Fix the task before using it on candidates.

## Test the rubric Give the rubric to two reviewers and ask them to score the same set of completed samples independently. Compare their scores and discuss every disagreement. A rubric with vague words such as professional or proactive will produce low agreement. Replace those words with observable criteria and examples. Keep the initial scores so the calibration conversation does not erase the disagreement. A rubric that cannot be applied consistently is not ready for a hiring decision.

## Look for adverse impact and unnecessary barriers A sample may be valid for the task and still create an unfair barrier. Check whether the instructions assume a tool, a time zone, a language variety, or a device that the role does not require. Check the time limit against the actual task. A candidate with a disability or a slower connection may need a different format. The goal is to measure the capability, not the candidate's access to the same setup as the interviewer. Document any accommodation and keep the scoring standard.

Discussion

## Compare the sample with later performance If the sample is used to hire, follow the first ninety days of work. Compare the sample scores with the reviewed work samples from the job. Do not expect a perfect correlation. Look for whether the sample predicted the specific construct it was meant to measure. If candidates who scored well on source verification still miss sources on the job, the sample may not have tested the real source environment. Use the result to revise the task, not to label the hire.

## Protect the candidates A work sample takes time. Keep it bounded, explain how the output will be used, and pay for substantive work that produces value. Do not reuse a candidate's solution as production work without an agreement. Store the sample and the score under a retention rule, and restrict access to the hiring team. A candidate who is not hired should not have their work circulated as an example. The process is part of the employer's reputation. Tell candidates in advance what tools and time they need, and offer an equivalent alternative when the default format would test access rather than ability.

Implications and limits

## Limitations A small hiring round cannot establish statistical validity. The sample measures performance under observation, which may differ from daily work. The rubric reflects the organization's current standard, which may change. A valid sample is one input among several, not a complete prediction. The method should be repeated when the role, the tools, or the work change. Where the sample is used across countries or languages, check that the instructions and the examples do not assume local idioms, currency, or software that the role does not require.

## Decision use Report the construct, the task, the rubric, the reviewer agreement, the accommodations, and the later performance check. If the evidence is weak, use the sample as a structured interview aid rather than a decisive score. If it is strong, keep the task and the rubric stable so candidates are compared on the same basis. The point is a fair, repeatable decision, not a clever test.

## Reporting Keep a short validity note beside the hiring record: what the sample measures, how it was scored, who reviewed it, what accommodation was offered, and what was learned from the first ninety days. Do not publish candidate scores or samples. The note helps the next hiring round avoid repeating a flawed exercise and gives the buyer a defensible account of how the decision was made. Review the note with the hiring manager before the next role opens, not after the shortlist is already set.