The teleoperation data flywheel

You cannot download more robot data. Every improvement therefore has to come from collecting better episodes, curating harder, or extracting more signal from what you already have.

Language models got good because the internet already existed. Robot policies have no equivalent. Every training episode was produced by a person moving a physical machine in real time, which puts a hard floor under the cost of a data point and makes the entire discipline of imitation learning downstream of one question: how do you manufacture demonstrations efficiently and well?

This is what we have learned about running that pipeline as a system.

The rig determines the ceiling#

Before any policy question, there is a hardware question: how does the human's intent get into the robot? The answer bounds everything after it.

The choice is not just about throughput. It determines what the data contains. Operators without force feedback produce demonstrations that mash; the policy learns to mash. Operators using a leader arm with a different mass distribution produce accelerations the follower cannot achieve, and the policy learns to command trajectories the robot then tracks with lag.

Every property of the collection rig becomes a property of the policy. Choose the rig for the artefacts you can live with, not for the throughput number.

Timestamps are the whole ballgame#

The single most common data bug we find, in almost every codebase, is misalignment between camera frames, proprioceptive state, and commanded actions. It does not throw an error. It produces a policy that is quietly 40 milliseconds behind reality and therefore slightly wrong at every contact event.

The discipline:

• Stamp at the source. Record the driver-provided capture timestamp, not the time your Python callback ran. The difference is your USB stack's mood.
• One clock. Every producer stamps against the same monotonic clock. If you have devices on separate machines, run PTP, and verify it — a clock offset that drifts over a session produces a dataset whose alignment error is a function of how far into the session an episode was recorded.
• Store raw, align on load. Do not resample at record time. Persist every stream at its native rate with its true timestamps, and do the interpolation in the data loader where you can change your mind later.
• Assert on load. A cheap check — maximum inter-stream timestamp gap per sample — catches the whole class of problem before it reaches a training run.

def align(episode, hz=30):
    """Resample every stream onto one grid at load time, with an explicit
    staleness bound. Raise rather than silently interpolating over a dropout."""
    t0 = max(s.timestamps[0] for s in episode.streams)
    t1 = min(s.timestamps[-1] for s in episode.streams)
    grid = np.arange(t0, t1, 1.0 / hz)

    out = {}
    for name, s in episode.streams.items():
        idx = np.searchsorted(s.timestamps, grid, side="right") - 1
        staleness = grid - s.timestamps[idx]
        if staleness.max() > 2.0 / hz:
            raise DataError(f"{name}: {staleness.max()*1000:.0f} ms dropout")
        out[name] = s.values[idx]          # zero-order hold; no invented frames
    return out, grid

Curation beats collection#

The instinct is to collect more. The higher-yield move, past a few hundred episodes, is usually to throw some away.

We score episodes on four axes and look at the joint distribution rather than any single number:

1. Success, obviously — but graded, not binary. "Succeeded after three attempts" is a different training signal from "succeeded first try".
2. Smoothness. Integrated jerk over the trajectory. High-jerk episodes are usually an operator fighting the rig, and they teach the policy to fight it too.
3. Duration relative to the task median. Both tails are suspicious. Much faster often means a shortcut that will not generalize; much slower usually means confusion that got resolved off-camera.
4. Coverage contribution. How much does this episode's initial state add to the set you already have? An episode from a starting pose you have sampled forty times is worth much less than the first one from a new corner of the workspace.

That last axis is the one teams skip and the one that matters most. A greedy farthest-point selection over initial states will tell you, concretely, where to send your operators next — and it usually contradicts intuition, because operators naturally gravitate to poses that are comfortable to reach.

The flywheel: intervention data#

The step that turns collection from a fixed cost into a compounding one is running the policy and correcting it.

Deploy the current checkpoint. When it is about to fail, an operator takes over, recovers, and hands control back. You record the whole thing, labelled with the handover boundaries. This is DAgger in its practical form, and it is extraordinarily efficient because every episode is collected in exactly the states the current policy actually visits — which is where its errors live.

Three implementation notes that make the difference between this working and not:

Blending, not switching. A hard cut from policy to human produces a velocity discontinuity in the recorded action stream, which the policy then learns as a legitimate behaviour. Ramp authority over 100–200 ms and record the blend weight alongside the actions so you can mask the transition.

Label the trigger, not just the intervention. Why did the operator step in — wrong object, wrong grasp, about to collide, too slow? That label is what turns a pile of interventions into a prioritized list of what the policy cannot do yet.

Weight interventions higher, but not by much. The corrected segments are the most informative data you have, and oversampling them by 2–3× is usually right. Oversampling by 10× produces a policy that has learned to behave like a recovering-from-failure policy at all times, which looks like nervousness.

def sample_weights(episodes, base=1.0, intervention=2.5, recency_halflife=30):
    """Interventions matter more; recent data matters more (the robot,
    the scene and the operators all drift)."""
    w = []
    for ep in episodes:
        x = intervention if ep.has_intervention else base
        age_days = (today - ep.date).days
        w.append(x * 0.5 ** (age_days / recency_halflife))
    return np.asarray(w) / np.sum(w)

Storage, versioning, and the boring parts#

A year of collection is tens of terabytes of video. The parts of the infrastructure that turn out to matter:

• Content-addressed episodes. An episode id that is a hash of its contents means "which episodes were in run 47" has an exact answer forever.
• Manifests, not directory scans. A training run consumes a frozen manifest of episode ids. Anything else and your dataset changes underneath you while you are trying to compare two checkpoints.
• Decode where the GPUs are. Video decode is the usual hidden bottleneck. Either pre-extract frames at training resolution or use hardware decode in the loader; a fine-tune that is 70% idle waiting on JPEG decode is a common and invisible waste.
• Keep the raw video. You will change your mind about resolution, crop, and camera set. Re-collecting is orders of magnitude more expensive than storing.

What good looks like#

A team running this well can answer, without a meeting: how many episodes exist per task and per initial-state cluster; which operator collected each one; what the intervention rate was for the current checkpoint last week; and which exact manifest produced the model currently on the robot.

None of that is machine learning. All of it determines whether the machine learning works.