History-conditioned MoT (no diffusion forcing): ground truth vs. rollout

The no-diffusion-forcing ablation (mot_hist_vt_actionimage): same two-expert MoT, action-image + cross-attention conditioning, actionflow prior, data, seed and optimizer as the DF action-image model — but trained as standard history-conditioned diffusion: the 2 history latent frames are exactly clean, all future frames share one noise level, the loss covers future frames only, and sampling denoises the whole future jointly (full_sequence). This page is its continued-training checkpoint: training resumed past epoch 100 with early stopping (patience 15 on val_loss), which stopped at epoch 144 with the best checkpoint at epoch 129 — shown here.

240.8M params Wan2.2 VAE latents, C=48, 16×16 checkpoint: epoch 129 (early-stopped continuation) no diffusion forcing

Where prediction starts: each row gives fps6 frames [start, start+8) — 2 latent frames — as clean context; prediction begins at fps6 frame start+8. In the decoded 13-frame clips the boundary is frame 5 (the context latents decode to f0–f4).

Ground truth vs prediction, frame by frame

Architecture in one paragraph

Both experts run in lockstep: each cross-attends its own action context, then their tokens are concatenated (256 video + 512 tactile = 768) for a single joint self-attention, then a per-expert FFN. The joint mask is temporally causal across modalities — camera frame t attends tactile frames ≤ t and vice versa. Each modality carries its own noise scheduler with independently drawn per-frame noise levels, so the two branches sit at unrelated points of their diffusion schedules. On the tactile side a frozen action-motion encoder reads the inverse-action displacement map on the gel plane plus a calibrated physics flow prior and injects a zero-initialised motion feature into the future tactile tokens; a frozen latent-flow probe scores the predicted clean latents against measured AllTracker optical flow.

The action, projected into pixel space

This is the half of the action the video expert cross-attends — action[0:18]. Both sensors' gel triads are projected into the middle camera and carried as absolute image coordinates: 2 sensors × 3 points × (u, v, z), with u,v normalised to [0,1] and z in metres. The tactile expert never sees this; it gets the body-frame SE(3) step instead.

Rendered at the 6 fps action grid over the 16 s long-rollout span; the short-window rows are the first 16 frames of these. The triads landing on the physical sensors in every frame is also the calibration check — a forward-project/triangulate round trip would return ~0 residual even with wrong extrinsics, so only the image can falsify it.

Metrics

Decoded-pixel PSNR (mean ± sem over n = 6 rollouts per cell), GT vs. rollout, through the Wan2.2 VAE — tactile streams via the tactile LoRA, the camera view via the base VAE. tac is the mean of the two sensors.

Against the diffusion-forcing action-image model

The DF model (mot_df_vt_actionimage, final epoch-100 checkpoint; its own Space has the clips) differs ONLY in the noising scheme. Same 12 rollout rows. For reference, this run's epoch-100 checkpoint (before the continuation) scored: short test 21.43/36.04, short train 20.94/35.41, long test 17.57/28.18, long train 16.80/27.76 (view/tac dB) — the continuation mainly helped long-horizon tactile (+0.9–1.0 dB).

Δ is no-DF − DF in dB; negative means diffusion forcing is better. DF wins every cell — including the short-horizon task the history model trains on directly.

Drift: how error grows over the 16 s rollout

Long rollouts are autoregressive — the model's own prediction becomes the next step's context, so error compounds. Per-latent-frame MSE, averaged over the 6 rollouts in each split, shaded ±1 sd. The first 2 frames (shaded) are ground-truth context, not predictions. Frame 0 is exactly zero; frame 1 is small but non-zero because the continuous GT reference is stitched across per-window VAE encodes while the rollout is seeded from a single encode — the two are both ground truth and differ slightly past frame 0. That offset is a property of the reference, not rollout error.

Short window — 2.7 s, 2 latent frames given, 2 predicted

Left half of each clip is ground truth, right half is the model's rollout.

Long rollout — 16 s autoregressive, predictions fed back

Browse every rollout

How it was run

RUN=.../runs_mot/mot_hist_vt_actionimage
CKPT=$RUN/checkpoints/best-epoch=129-val_loss=0.2249.ckpt   # early-stopped continuation

# short window (one 16-frame window; 2 latent frames of context)
python -m vm_diffusion.scripts.infer_windows --ckpt $CKPT --config $RUN/config.yaml \
    --windows motherboard_0510_episode_005:0 ... --out_dir $RUN/eval/short_test

# long rollout (sliding window, keep-1, driven by the episode's real action sequence)
python -m vm_diffusion.scripts.long_rollout_mot --config $RUN/config.yaml --ckpt $CKPT \
    --episode motherboard_0510_episode_005.pt --start 0 --seconds 16 --fps 6

# decode both through the Wan2.2 VAE (tactile LoRA for tl/tr, base VAE for view)
python decode_rollout_wan22.py --rollout-dir <cell dir> --fps 6

Caveats

• Six rollouts per cell (two episodes × three start offsets). Treat the sem as indicative, not as a significance test.
• This is a held-out-episode split, not a generalization split. The test episodes are the same board and the same session as many training episodes, so these numbers measure "unseen trajectory, seen conditions".
• PSNR rewards blur. A model that predicts a smooth mean-ish gel image scores well. Watch the clips, not only the table — the tactile texture is where the difference shows.
• The checkpoint is last.ckpt, not best-*. Validation loss for this model is ~56% the frozen-flow-probe term, which oscillates and does not track denoising quality, so selecting on it would pick a checkpoint for the wrong reason. A fixed step count is also the fairer comparison against the baseline.
• Long-rollout GT reference is stitched across per-window VAE encodes, so it differs slightly from any single encode past frame 0. The rollout seed itself is exact.
• The projected action is not always in frame. Across these rows the gel centre is in-frame 100% of the time for most, but as low as 88.5% (train row 0). When a sensor leaves the frame its action image is blank, so the video expert receives no positional signal for that step — while the tactile expert's body-frame SE(3) half is unaffected. The two halves of the action degrade independently.

Diffusion forcing (per-frame independent noise levels), pyramid sampling with 100 denoise steps, 36-D causally-masked action conditioning, 3D axial RoPE. Latents are Wan2.2 VAE, 48 channels on a 16×16 grid at 6 fps.