Independent review
Apply the same guideline version to separate judgments.
Adjudication
Record why labels differ. Correct the answer or clarify the rule.
Batch decision
Use agreed checks to accept the work or return it for rework.
An accepted batch is not a guarantee of model performance. Preserve known errors and review rationale with the delivery.
Layer 1 — Gold set
A fixed set of items with adjudicated answers, used to qualify annotators and seeded invisibly into production queues.
Layer 2 — Agreement
Overlap a percentage of items across annotators and report a chance-corrected coefficient. Krippendorff's alpha handles multiple raters, missing data and different measurement levels (K. Krippendorff, University of Pennsylvania (ScholarlyCommons)). Report it per class, not just overall.
Layer 3 — Adjudication
Disagreements go to a senior reviewer with a written rationale. The rationale becomes guideline material.
Layer 4 — Error taxonomy
| Error class | Example |
|---|---|
| Omission | Object present but unlabelled |
| Commission | Label on nothing |
| Class confusion | Wrong category |
| Boundary | Box or segment edges imprecise |
| Guideline gap | Case not covered by rules |
Layer 5 — Batch acceptance
- Define acceptable error rate per class before work starts.
- Sample size set statistically, not by convenience.
- Reject-and-rework rules written into the contract.
Have an annotated dataset or specialist expertise?
Tell us what you own, how the labels were reviewed and which rights you can license. A short overview of your dataset or annotation business is a useful place to start.
Prefer to write directly? hello@callwavemedia.com