This is a useful problem. One thing I’d want to see is a clear split between corrupted data and unusual but valid demonstrations, since rare movements can still be valuable in real-world data. A few visual examples for each failure type would make the tool much easier to trust.
The corrupted-versus-unusual-but-valid distinction raised upthread is the whole problem, and it has a measurable form: what is Calibra’s false-positive rate on a dataset you know is clean?
Concretely. Take a LeRobot dataset you are confident in, run the detector, and publish the per-check flag rate. Jittery motion and calibration drift will both flag on some genuinely good demonstrations, because a human demonstrator is jittery and a real rig does drift. That rate is not a bug to tune to zero; it is the number a user needs in order to decide what to do with 40 flags on their own data.
The reason this matters more here than in most tooling: a detector at the dataset layer sits upstream of every policy trained on it. If it silently removes 8 percent of valid demonstrations on a particular task family, the downstream success rate moves and nobody attributes it to the cleaner.
Same discipline we hold ourselves to one layer down, at policy evaluation: every detection rate gets published beside its firing rate on known-benign input, and if the firings concentrate on a subset rather than spreading, that is a signal about the detector, not the data.
Picking up the point about where the reporting minimum should live.
I have come round to thinking it belongs in the dataset card, not in the tool. A tool-side check can be skipped and leaves no trace. A card field is visible to the next person who downloads the dataset, whether or not they ran anything.
Three fields would cover most of what I keep having to reconstruct by hand:
episodes_total and episodes_used, separately. Almost every dataset I have looked at has a filtering step, and the filtered count is what a success rate is actually divided by. Right now that denominator is usually not recoverable.
simulator and its version, or hardware and the platform. Not both, one or the other, and stated. A dataset that does not say which it is cannot be compared with one that does.
known_defects, free text, allowed to be empty but not allowed to be absent. The forcing function is having to write “none found”, which is a different claim from silence.
What I am not proposing: a quality score, a badge, or anything that ranks datasets. Those invite gaming and I do not think anyone wants to maintain the rubric.
Our own cards do not fully do this yet, so this is a proposal I am also behind on.