Glossary
Working definitions used across this desk.
- Adjudication
- Resolution of disagreement between annotators by a designated senior reviewer, ideally with a recorded rationale.
- Annotation schema
- The set of label types, classes and attributes a project uses, with a version number.
- Calibration set
- Items with agreed answers that annotators complete before production work to show they understand the guidelines.
- Chance-corrected agreement
- An agreement statistic that discounts agreement expected by chance, such as Krippendorff's alpha or Cohen's kappa.
- Contamination
- Exposure of evaluation items to model training data, which inflates measured performance.
- Datasheet
- A structured document describing why and how a dataset was made and how it may be used.
- De-identification
- Removal or transformation of identifiers so data no longer relates to an identifiable person under a given legal standard.
- Evaluation-only licence
- Permission to use data to test models but not to train them.
- Expert annotation
- Labelling that requires professional qualification or specialist knowledge to perform correctly.
- Gold set
- Items with adjudicated answers used to qualify and monitor annotators.
- Interpolation
- Automatic generation of labels between manually set keyframes in video.
- Managed labelling
- A service where the vendor supplies workforce, tooling and QA under contract.
- Pre-labelling
- Model-generated draft labels that humans correct.
- Preference data
- Human judgements comparing two or more model outputs.
- Provenance
- The documented origin and chain of custody of data and labels.
- Work-product assignment
- Contract clause transferring rights in created material, such as labels, to the client.