Robot learning · Bonn · updated 12 September 2026

Where does the robot touch the object?

We record exactly that, frame by frame, from five robot simulators, so a model can learn actions from it.

simulated scene→contact flow→actions

real video→contact flow→actionslater

Real video has no ground truth for contact. The simulators do, so they give the video half of the chain something to be scored against.

LIBERO, one recorded grasp replayed in MuJoCo. Every dot is one contact point the physics engine reported in that frame, with a short line along its surface normal.

What one contact is

Five facts per touch, hundreds of touches per episode

A physics engine already knows every point where two things press together. We write those points out instead of throwing them away, and follow how each one moves from frame to frame. That is contact flow.

It is not the total squeeze force, and it is not optical flow. Which piece of an object is touched matters: a fingertip on the cap and a fingertip on the barrel are different grasps.

What one episode contains

Four views of the same frame, one file

Colour render of a LIBERO scene — Colour image

Depth image of the same frame — Depth, metres per pixel

Segmentation of the same frame — Segmentation, one colour per object

Contact points projected onto the same frame — Contact points, from the physics engine

The 23 arrays every episode file must carry

3 images: colour, depth, segmentation 2 camera matrices 3 object tables: names, poses, ids 2 robot arrays 13 contact arrays, down to the geometry piece

An episode counts as delivered only when all 23 are present. A checker reads each file and says done or stale; stale files are re-rendered, never skipped.

The five simulators

Each with the robot it ships with

LIBERO

MuJoCo

Franka arm, tabletop pick-and-place. Recorded human demos replayed.

renders full episoderuns on Marvin

ManiSkill

SAPIEN

Franka arm, cube tasks. Scripted grasps.

renders full episoderuns on Marvin, lmgpu

ABC-Sim

MuJoCo

Two-arm YAM rig, bottle task. One shipped scene so far.

renders full episoderuns on Marvin

BEHAVIOR

Isaac Sim

Wheeled two-arm household robot. 51 furnished houses, 1,018 tasks.

renders full episoderuns on Marvin

GenieSim

Isaac Sim

AgiBot G1 dual-arm robot. Pick, pour and workpiece tasks.

renders full episoderuns on Marvin

The first three clips are proof episodes from our workstation. The GenieSim clip is a delivered episode, with that frame's contact points drawn on the head camera. The BEHAVIOR clip still comes from an earlier run in a neighbouring project.

How an episode gets made

One command, no screen, no driver fiddling

Anyone here can ask for N episodes from a named simulator on a named cluster. All six simulators render this way on Marvin; a MuJoCo episode takes seconds, a GenieSim one minutes, a BEHAVIOR one about a GPU-hour.

Where we are

The phase is twenty episodes from each simulator

7,652episodes on disk, twenty or more from every simulator

Every episode passes the 23-key check. The renderers now record every contact between the robot and its world. They also record the pose of every robot link in every frame. ManiSkill was re-rendered with them on 10 September, REALM and LIBERO were re-rendered the same day.

simulator

renderer

what moves the robot

delivered

latest

LIBEROMuJoCo

23 keys

human demos

5,288 / 20

Re-rendered 10 September with the gripper pads and arm links as tracked bodies, straight into the compact format: 9.4 GPU-hours, 12 GB. 1,212 replays refused by name for leaving the recorded path.

ManiSkillSAPIEN

23 keys

motion planner

234 / 20

Re-rendered 10 September with every robot contact. Contact rows up 23 percent. The robot on its table was 0 rows before and 121,113 now. Six planner refusals named.

ABC-SimMuJoCo

23 keys

scripted grasp

20 / 20

20 delivered from the one shipped scene.

BEHAVIORIsaac

23 keys

human demos, replayed

187 / 200

187 on disk. The 13 missing episodes are queued on Marvin.

GenieSimIsaac

23 keys

scripted grasp, GenieSim's own IK

20 / 20

20 delivered, scripted grasps on five ICRA tasks. Its recorded set replays too: 8 of a batch of 30 came back as full episodes.

REALMIsaac, OmniGibson

23 keys

DROID policy or scripted pilot

10 + 20 + 40

10 pilot, 20 policy, 40 perturbed. Re-rendered 10 September with the robot's link poses. The policy completed 22 of 60 rollouts this time, 16 before.

Where the files are

Marvin holds every cluster-rendered episode, 65 GB in all after the 10 September conversion to compressed video streams. It was 2.8 TB. The workstation holds three proof episodes. One older pilot file at a partner lab has 20 keys and is stale.

Found on our storage, 9 September

GenieSim ships its own recorded set: about 129,000 episodes, with every robot joint but no object poses. Our renderer replays it on the shipped layouts, and 8 of a batch of 30 came back as full episodes. InternData-A1's Isaac assets are here too.

Proven live

The shared Isaac contact reader ran on a replayed BEHAVIOR demonstration and inside a headless GenieSim task. Both runs showed that a sleeping object reports nothing at all.

Six steps, all done, and the next phase has started

1Render LIBERO and ManiSkill on Marvin. Done 6 September.done

2Build the ABC-Sim environment on a cluster and render it. Done 6 September, 20 episodes.done

3First live Isaac contact, on a replayed BEHAVIOR demonstration. Done 7 September.done

4Convert the GenieSim image and write its first full episode. Done 7 September.done

5Stage both Isaac simulators on Marvin. Done 8 September.done

6Ship the contact table to the partner lab. Done 9 September, all five simulators.done

Next phase, started 9 September

Use it, then scale it

AContact-flow videos that draw only the flow. Done 9 September: hue is the direction a contact moves, saturation is its speed.done

BCan GenieSim replay its own recorded dataset? Yes, 9 September: 8 of 30 written, every refusal measured.8 of 30

CThe Isaac contact tables. Done 9 September, all five simulators checked the same way.done

DWhere the trajectories come from. Done 9 September: no sixth simulator was needed until REALM, which step J adds.done

ELIBERO, the whole benchmark. Done 9 September: 130 tasks.done

FManiSkill, all twelve planner tasks. Done 9 September.done

GCan a colleague ask for episodes from a fresh shell? Tested 9 September: three commands worked, and the help text was fixed where it misled.done

HLIBERO, all fifty demonstrations of every task. Done 9 September: 5,314 episodes.done

IBEHAVIOR, twenty demonstrations of each of ten tasks.200 of 200 are on disk since 11 September. The last 13 carry the robot's link poses; the other 187 do not. An episode costs about 1.3 GPU-hours; the set is 56 GB compressed.200 / 200

JREALM, the DROID simulator.All 60 policy episodes were re-rendered 10 September with the robot’s link poses. The DROID-trained policy completed 22 of 60 rollouts this time, 16 before.policy 16 of 60

KEvery contact between the robot and its world.Contact flow is every contact the robot makes, not only the gripper on the task object. Two gaps are now closed: ManiSkill threw away contacts that had no task object, and no simulator wrote down where the robot’s own links were each frame. REALM’s contact cap was also raised, so busy frames no longer drop furniture-on-floor rows. REALM also renders 17.2 episodes a GPU-hour, against 10.7 before.landed

LTwo more simulators: RoboCasa kitchens and RoboTwin bimanual tasks.Both render into the same format since 11 September. RoboCasa replays human demonstrations from their recorded state, 7 seconds an episode, 55,000 available: the first 1,800 human episodes over 18 tasks are on Marvin since 12 September, none refused, 9 GB. RoboTwin generates fresh scenes, 16 seconds an episode, no ceiling; its 50-task pilot is queued. ABC-Sim now also runs its released policy: five times the grasp contact of the scripted driver at 17 times the cost.started

LIBERO. Only contact flow is drawn. The grasped can turns violet on the lift and yellow when the arm reverses over the basket. The colour wheel is the legend.

BEHAVIOR. The radio in the right gripper shows red and blue at once while it rotates, because its two ends move in opposite image directions. It turns magenta as it is set down.

GenieSim, a recorded stock-the-shelf episode replayed on the shipped scene. The bottle's contacts colour as it is lifted from the shelf and carried.

REALM, the DROID rig. The DROID-trained policy picks up a teaspoon under a lighting perturbation. Only moving contacts are drawn. The re-render with the robot's link poses is done.

Decisions we need from you

None of these block the first twenty

BEHAVIOR: do we want its tasks or its houses? Decides what the second batch looks like.

Tell the partner lab their row key is incomplete? Without a contact-slot field a join silently drops more than half the hand-object rows.

Does the pilot episode ship as it is, with one check marked satisfied but uninformative?

Re-render the 200 BEHAVIOR episodes with the robot's link poses? About 260 GPU-hours and 2.4 TB, what the first render cost.

Render LIBERO's wrist camera for the lab? It answers their head-camera request better than the fixed view.

A dexterous hand later, as its own column? Everything here is a parallel gripper.

Words used above

Episode: One recorded task attempt, a few hundred frames, saved as one file.
Contact impulse: The push delivered through one contact point in one frame. Summing them gives force; we keep the points.
Surface normal: The direction perpendicular to the surface at the contact, the direction the push acts along.
Geometry piece: Objects are built from several collision shapes. The piece says which one was touched: cap or barrel.
Headless: Rendering on a machine with no screen attached. Simulators usually assume a screen; clusters have none.
Cluster, Slurm: A shared pool of GPU machines with a queue in front. Marvin, lmgpu and CLAIX are the ones we use.
Container, Apptainer: A sealed copy of a program and everything it needs, so it runs the same on any cluster.
Isaac Sim: NVIDIA's simulator, built on the PhysX engine. Used by BEHAVIOR and GenieSim.