Skip to content

Robotics annotation: labelling actions, not just pixels

Robot learning data pairs sensor streams with actions and, increasingly, natural-language instructions. The label is often the instruction and the success flag.

Read the guide
Sequence & context → Aligned annotations → Reviewed transitionsThree connected layers illustrate how sequential source context is preserved in aligned labels and reviewed transitions.
Sequence & contextAligned annotationsReviewed transitions

What a robotics episode contains

  • Sensor streams: RGB, depth, proprioception, sometimes force/torque.
  • Action stream: joint or end-effector commands at a fixed rate.
  • Language annotation: the instruction or a post-hoc description of the task.
  • Outcome label: success, failure, or partial, often human-judged.

Cross-embodiment pooling

The Open X-Embodiment collaboration pooled datasets from many robot types into a shared format (Open X-Embodiment Collaboration, arXiv:2310.08864). For owners, this signals that format conversion and per-robot metadata (kinematics, control frequency, camera placement) are part of what makes data reusable.

QA specific to robotics

CheckWhy
Instruction matches observed behaviourPost-hoc captions drift from what happened
Success label criteria written down'Success' varies by operator
Timestamp alignment across sensorsMisalignment silently corrupts training

Rights notes

Teleoperation data recorded in homes or workplaces may capture people and private spaces. Treat it like video for consent purposes, and confirm that operators' contracts cover their demonstrations.

Have an annotated dataset or specialist expertise?

Tell us what you own, how the labels were reviewed and which rights you can license. A short overview of your dataset or annotation business is a useful place to start.

Prepare for a licensing conversation