This page explains what the project is building, why it matters, how the pipeline works end to end, and exactly where things stand today. No prior knowledge assumed.
The project is building a chain: a simulated scene produces contact flow, and contact flow trains actions. Later, the same chain runs on real video instead of a simulation.
Contact flow means where on an object each contact happens, and how that spot moves over time — the position of each contact point, the direction it pushes (its surface normal), and the strength of the push at that instant (its contact impulse, a short burst of momentum, not how hard the gripper is squeezing overall — the total squeeze force is a side effect, and it's a different thing from optical flow, which is just pixel motion in a video).
Simulators exist to give the video half of the chain something to be scored against. For a real video there is no way to know the true contact point at all; a simulator's physics engine can report it exactly, so it becomes the answer key the video model is checked against.
Each of the five simulators — GenieSim and BEHAVIOR on Isaac Sim, LIBERO and ABC-Sim on MuJoCo, ManiSkill on SAPIEN — is a physics engine plus the robot it ships with. A simulator here is software that runs real physics on a virtual robot and scene instead of a physical one, so an episode can be recorded without any hardware in the loop.
Every run is headless: it renders images and runs physics with no monitor attached, because it executes on a shared cluster machine reached over the network. One episode of one task becomes one .npz file — a compressed array archive — carrying a fixed 23-key recording, grouped into three parts.
An episode only counts as done once every one of the 23 keys is present — a status check reads each file's keys and reports it done, in progress, or stale. Nothing is taken on trust.
The current phase asks for twenty episodes from each of the five simulators — a hundred episodes total. That is the whole target for this phase; nothing further is being asked yet.
The honest count is zero delivered on all five simulators — but the thing that was blocking every one of them opened today. Until today, no engine reliably named which piece of an object a contact sat on. Now the geometry-piece column exists, and was checked against a real run, on all three non-Isaac engines: LIBERO already had it, ABC-Sim was assumed to have it for free and did not, and ManiSkill needed a brand-new way to identify a piece at all, because its physics engine (SAPIEN) gives shapes no name or id of their own — a piece there is now identified by its body plus its position in that body's own shape list.
Five earlier working branches were also merged today, so the whole project now lives on one tree, and four dead scripts and two stale planning documents were deleted along with it. Isaac's own contact reader has been written and reviewed on paper, but it has still never been executed.
Everything recorded before today is stale by the new schema — the roughly 600 old ManiSkill episodes (605 counted by the key checker, 900 by the nominal job size) carry 11 keys, the 25 old LIBERO episodes carry 17 — and all of it will be re-rendered rather than resumed, because the status check classifies each file by the keys it actually has.
Everything above describes intent — what should be true once the current push lands. Today, two separate checks went underneath that: one read every script in the repository line by line, the other logged into beech and every cluster and looked at what is actually sitting on disk and in the job queues. Where the plan and what was found disagree, this section follows what was found.
Twenty episodes from each of the five simulators — a hundred delivered this phase.
Zero episodes sit at any cluster's output folder today. Three episodes carrying all 23 keys exist — one per working renderer, rendered on beech earlier today. The pilot already handed to the reproduction lab carries 20 of the 23 keys. Renders made before today's format carry only 11 keys (ManiSkill) or 17 (LIBERO), sitting in old paths, and none of the above count toward the hundred.
source: disk, cluster statusEach of the five simulators has a script that writes the full 23-key recording.
Three scripts do this, for LIBERO, ManiSkill and ABC-Sim. For GenieSim and BEHAVIOR there is no renderer file at all — nothing to run yet. What exists instead is a physics-contact reader shared by both, and a GenieSim probe script; both are written and neither has ever run against a live physics engine.
source: codeManiSkill is ready to run on Marvin, lmgpu, and lmdort.
Marvin and lmgpu each have a working software environment for ManiSkill, confirmed live today. lmdort does not: the environment the code expects is missing from its disk, and its own working folder refuses to even create a directory — permission denied. No cluster, Marvin included, has an Isaac asset folder set up yet, so neither Isaac-based simulator can run anywhere, independent of GPUs.
source: code, clusterA GenieSim container, an Isaac Sim image, and the BEHAVIOR container with its scene assets are already staged.
Confirmed on disk: the GenieSim container image (37.9 GB), an Isaac Sim 5.1 image, the BEHAVIOR container (14.5 GB) with 36 GB of its own scene assets, one of the twenty BEHAVIOR demo recordings the next step needs, and two separate copies of the GenieSim asset library (24 GB and 27 GB).
source: disk4.8 terabytes free on the shared data drive.
That number holds: /home/data has 4.8 TB free out of 11 TB. But /home/user, where most of this project's code and Python environments actually live, has only 171 GB free out of 1.8 TB — 91% already used.
Zero GPU-seconds spent in this phase so far.
True, and nothing of this project's is waiting either. Marvin has one unrelated job waiting for priority; lmgpu has three unrelated jobs running or waiting; lmdort's queue is empty. The one GPU on beech itself sits idle.
source: clusterThe map below shows physical machines, not code — what is actually installed, downloaded, or written to disk on each one, checked today.
Almost everything below can start right now, in parallel. Only two real dependencies hold anything back.
Nothing blocks this today. Submitting both on the Marvin cluster costs roughly one GPU-hour of compute together; the real wait is queue time. run_sim.py submit --simulator libero --cluster marvin --episodes 20, then the same for ManiSkill. Twenty per task, three tasks each, comes to sixty episodes rendered. Done means the status check reports every key present.
The renderer already works, proven on the development machine. It just needs its software environment built on one cluster with no GPU required for that step, then the same submit command. Twenty episodes of the bottle task it already ships satisfy this phase; new scenes come later.
Download the nineteen still-missing demonstration recordings (about 400 MB), then run the contact reader against one of them, replayed inside the existing container. This is the step that proves the physics-contact reader GenieSim will reuse unchanged — so it pays for both Isaac-based simulators at once.
Convert its built container image to the format the cluster runs, then run its probe script, written but never executed, for one task, headless, to completion. No GPU is needed for this conversion step.
Stage each one's asset files and a writable cache on the Marvin cluster. This is the step that turns steps 3 and 4's single proof runs into twenty full episodes each.
Already ran end to end today, on a real LIBERO episode: 5,013 rows, none invalid. The one-off checks built for the pilot data were removed as no longer needed; a proper large-scale physics quality check would be new work, not yet started.
Slurm cluster — a shared pool of computers with scheduling software that queues up jobs and hands each one a machine, rather than everyone fighting over the same box.
Apptainer — a container format: a sealed box holding a program and everything it needs, so it behaves the same way on any cluster it is copied to, without reinstalling drivers or libraries by hand each time.
These do not block the current twenty-per-simulator phase, except where noted. They decide what happens next.
Is BEHAVIOR wanted for its huge library of tasks, or for its scenes? It has 1,018 task definitions set inside 51 furnished houses.
Doesn't block the first twenty episodes — it decides what the second batch looks like.
Should the reproduction lab be told their episode format is missing a field? Their row key needs a slot number for which contact point on a surface, and their object id currently only captures the first of several named pieces an object has. Without the slot number, a join silently drops 494 of the pilot's 850 hand-object rows.
The lab is building against this format right now. The working assumption is: tell them, and soon.
Does the existing pilot data ship as it is? It passes its physics sanity check, but that check isn't very informative yet.
The working read is: ship now, get a more meaningful check later once more complex scenes exist. Needs confirming, or a "don't ship yet."
Where is the real 8.8-terabyte ABC-130K asset copy? The path on record turned out to be a 23 GB working directory instead.
Not needed for the bottle task already shipped; needed for building further scenes with the real objects.
Should LIBERO's wrist-mounted camera be rendered for the lab? It exists, and answers their request for a head-mounted camera better than the current fixed camera does.
A cheap re-render — but the lab should be asked first.
Is a dexterous, multi-fingered hand wanted later, as its own column? Everything in this project currently uses a simple two-finger gripper.
Decides nothing in this phase.