Philippines staffing research · Updated
Does quality-review interpretation drift between calibrations?
Track independent scoring, disagreement direction, rubric versions, and adjudication.
Research question: How does reviewer agreement change between calibration events under a stable rubric?
Scope: eight cycles in one program with one eligibility rule and predefined dimensions.
Methodology: Assign overlapping blinded samples and retain independent first scores, rubric version, confidence, adjudication, and corrections. Separate targeted from random samples.
Unit of analysis: one independently scored case-dimension pair. Stable identifiers prevent messages and edits from inflating the denominator.
Classification: agreement, severity disagreement, requirement disagreement, insufficient evidence, rubric ambiguity, conflict, or unclassifiable. Observable states remain separate from causal judgment.
Measures: agreement proportions and denominators, direction, reviewer-pair coverage, time since calibration, and correction stability. Report counts, denominators, distributions, and missingness.
Validation: Use a reserved reference set and separately recode a stratified sample without prior scores.
A Philippines-based specialist may collect permitted records and apply the frozen codebook. Internal owners retain privacy, security, legal, financial, employment, policy, and customer-remedy decisions.
Use named access, minimum necessary data, documented retention, and de-identified reporting where practical. Record exclusions before outcome review.
Inference limits: Patterns can locate unstable interpretation but cannot establish reviewer competence or justify employment decisions.
Limitations: Reference cases may miss live complexity, reviewer pairs can be sparse, and adjudication is not infallible.
Conclusion: Clarify rubric language where reproducible disagreement clusters, then test new cases.