Sourcing grid
| Source | Best for | Watch for |
|---|---|---|
| Open crowd | High-volume, low-ambiguity tasks | Quality variance; redundancy needed (Snow et al., EMNLP 2008 (ACL Anthology D08-1027)) |
| Managed vendor | Stable, well-specified tasks at scale | Guideline drift across shifts |
| In-house team | Confidential or fast-changing tasks | Fixed cost, hiring time |
| Expert network | Judgement-heavy specialist tasks | Credential verification, availability |
Workforce conditions are a diligence topic
Buyers and regulators increasingly ask how data was produced. Document pay basis, working conditions disclosures provided by vendors, and whether annotators were exposed to harmful content and supported. Ask for written disclosures that explain the conditions behind your own dataset or annotation delivery.
Hybrid patterns
- Crowd first pass, expert adjudication of low-agreement items.
- Model pre-labelling with human correction — measure correction rate, not just throughput.
- In-house guideline owners managing external vendor capacity.
Have an annotated dataset or specialist expertise?
Tell us what you own, how the labels were reviewed and which rights you can license. A short overview of your dataset or annotation business is a useful place to start.
Email us about your dataset
Prepare for a licensing conversation Prefer to write directly? hello@callwavemedia.com