A public reference point
LegalBench assembled legal reasoning tasks with contributions from legal professionals and scholars, organised by reasoning type (Guha et al., arXiv:2308.11462). It illustrates a useful principle: evaluation items should be categorised by the skill they test, not just by area of law.
Task taxonomy for a legal eval set
- Issue spotting: identify which legal issues a fact pattern raises.
- Rule recall: state the governing rule for a jurisdiction and date.
- Rule application: apply a rule to facts and reach a conclusion.
- Interpretation: construe a clause or statute.
- Drafting critique: identify defects in a draft provision.
Confidentiality and privilege
Documents from client matters are typically bound by professional confidentiality duties and may be privileged. Synthetic fact patterns written by practitioners avoid most of these issues and are often the cleaner asset to license. Firms considering licensing should involve their professional-responsibility counsel.
What a buyer will ask
| Diligence item | Evidence |
|---|---|
| Author qualification | Bar admission or equivalent, verified. |
| Answer key review | Second practitioner sign-off per item. |
| Jurisdiction and date stamp | Each item tagged; law changes. |
| Contamination risk | Items never published online. |
Have an annotated dataset or specialist expertise?
Tell us what you own, how the labels were reviewed and which rights you can license. A short overview of your dataset or annotation business is a useful place to start.
Prefer to write directly? hello@callwavemedia.com