Why robotics data started looking like YouTube

What started as a robotics data platform ended up looking a lot like YouTube system design. In a post published 2026-07-20, Kingston Kuan explains how Pareto, Hebbian Robotics' open-source platform for robotics datasets, converged on the same core pattern: upload → process → store → index → play.

The original problem was narrow but demanding. Robotics teams wanted to upload LeRobot datasets, search across demonstrations, inspect synchronized camera feeds and robot states, and export curated episodes for training. Once the infrastructure work began, Kuan writes, the requirements became familiar: reliably upload large media files, preserve originals, process them in the background, generate thumbnails and previews, index metadata, and stream data to a browser without requiring a full download.

How robot episodes differ from ordinary video

A standard video platform revolves around a single main object: a video file, sometimes with audio and subtitles. A robot-learning dataset is more complex. One episode can include multiple camera feeds, joint positions, actions, force readings, timestamps, task descriptions, and other sensor streams.

Those video streams are only useful when they stay aligned with what the robot was sensing and doing at that moment. Pareto is built to search across episodes so a user looking for an “orange block” can inspect the surrounding trajectory, compare camera views, and decide whether a demonstration belongs in a training set.

Kuan points to LeRobot v3 as a concrete example. Episodes can share Parquet data files and MP4 shards, which means reading an episode requires resolving both its structured row range and its timestamp range inside each camera video.

Recommended reading

South Africa mining automation may widen inequality

Uploads, pipelines, and long-running jobs

For uploads, Pareto separates file receipt from later conversion and indexing. Source files are stored under versioned storage paths, completed batches are checkpointed, and the final manifest is written last. Downstream jobs treat that manifest as the completion marker, which helps avoid guessing whether a directory is complete and preserves traceability back to the exact input batch.

The processing layer then fans out into multiple generated assets, including:

• sampled frames
• thumbnails
• preview videos
• downsampled state and action series
• vector embeddings
• search indexes
• Rerun recordings

Pareto runs these as durable workflows using Temporal, which Kuan notes offers both a hosted service and an open-source, self-hostable option. Workers are split by capability and job shape. Independent episode ranges are sharded across GPU embedding workers, then merged at a finalization step. Other jobs are chunked to run across elastic GPU or CPU pools.

Not every artifact is generated up front. Kuan says Pareto moved many lightweight, recoverable computations to lazy, on-demand paths so the critical path stays focused on making data searchable and viewable. That shortens time-to-first-use and avoids spending resources on outputs no one has requested.

For live text search, the company built a CPU SigLIP 2 server using Rust and ONNX. The idea is to reserve GPUs for ingestion-heavy batch work while handling online text embeddings on CPUs.

Storage, retrieval, and synchronized playback

Pareto also mirrors another classic video-platform split: it does not put media, metadata, and search indexes into one database. Instead:

• source datasets and derived data live in URI-addressed object storage
• Postgres stores metadata
• LanceDB stores multimodal vectors for retrieval
• the server is stateless

A text query is embedded, matched against vectors, and returned as episodes.

On the frontend, Pareto streams only the bytes the viewer needs rather than opening the full original dataset for every interaction. The browser UI uses previews, supports seeking, and keeps multiple camera feeds on one synchronized transport while showing corresponding state and action data. Users can also pin preview quality across the camera grid.

Kuan argues that scale is not the immediate point. The more useful early decisions are structural: where durable state lives, whether work can be retried, and whether more capacity can be added without redesigning the system. At the same time, he warns against overbuilding. Pareto’s direct CLI can index, search, and export synchronously, without Postgres or Temporal; distributed components are added only where long-running, durable execution is necessary.

His bottom line: robotics data is not just video streaming with extra columns. But as a design shortcut for upload, processing, storage, indexing, and playback, the YouTube analogy gets surprisingly far.