Dimios45/yam-duster-bspline-dp

YAM Duster-in-Box — B-spline Diffusion Policy

A B-spline Policy (BSP) UNet diffusion policy trained on Dimios45/yam_duster_in_box for a single-arm I2RT YAM robot: "pick up the duster and put the duster in the box."

Instead of predicting a fixed grid of future actions, the policy predicts B-spline knots and control points. The result is a continuous trajectory that can be resampled at any rate and temporally rescaled — so a single prediction covers ~1.1 s of motion, and playback speed becomes a deploy-time knob rather than a retraining decision.

A plain Diffusion Policy baseline trained on identical data is at Dimios45/yam-duster-dp.

Action space — read this first

Actions are joint-space targets, not end-effector poses:

[right_joint_1..6 (radians), right_gripper (0 = open, 1 = closed)]

This differs from the upstream B-spline Policy YAM example, which uses EE pose + rot6d and solves IK at deploy. Here the policy commands joints directly, so no IK is involved. The gripper convention (0 = open, 1 = closed) matches constants.py in the upstream repo, so YAM_GRIPPER_INVERT handles the i2rt-side flip as usual.

The network output is shaped (16, 8): column 0 is the knot vector (in units of 10 Hz frames, relative to the current observation), columns 1–7 are control points for the 7 action dims. 16 = chunk_size 10 + 2 × degree 3.

Observations

key	shape	notes
top_image	(3, 128, 128)	RGB, resized from 640×480 (plain squash, no crop)
wrist_image	(3, 128, 128)	RGB, right wrist camera
joint_pos	(7,)	measured joints + gripper

Two observation steps (n_obs_steps: 2). Images are normalized to [0,1]; a random crop to 116×116 is applied in training and a center crop at eval.

The lowdim key must be named joint_pos. The dataset's get_normalizer only accepts lowdim keys containing pos/quat/qpos and raises unsupported lowdim key otherwise.

Files

file	size	use
deploy_ema.ckpt	426 MB	Inference. EMA weights only — what you copy to the robot.
epoch0600_full.ckpt	1.5 GB	model + ema_model + optimizer, for resuming or fine-tuning.

Both embed the full Hydra config (pickled with dill), so the policy rebuilds itself on load — but bspline_policy and diffusion_policy must be importable, since cfg._target_ is a class path.

Training

base	B-spline-policy/bspline-policy UNet BSP
data	50 episodes, 37,986 frames @ 30 Hz → resampled to 10 Hz (12,677 steps → 12,627 chunks)
hardware	1× RTX 4090, 2 h 04 m, 4.1 GB VRAM, ~16 it/s
epochs / batch	601 / 64
optimizer	AdamW, lr 1e-4, cosine, 500 warmup steps, EMA
params	66.9M diffusion + 22.4M vision (ResNet18 ×2)
scheduler	DDIM, 100 train timesteps, 16 inference steps, epsilon prediction
B-spline	degree 3, chunk_size 10, max_error 0.002 rad, absolute knots
final train_loss	0.002

30 → 10 Hz resampling is deliberate: the YAM stack runs at POLICY_CONTROL_FREQ = 10, and it makes one 16-knot chunk span ~1.1 s instead of ~0.35 s.

Measured behavior

Predicted chunks decoded through the real deployment path and resampled at 100 Hz, over 200–300 held-out samples:

metric	value
open-loop arm error vs demos	median 0.81°, p90 1.68°
error tail	p99 28°, max 63° (3.7% of chunks > 10°)
chunk duration	1.10 s predicted vs 1.10 s demo-fit
peak arm velocity @1×	48 °/s p95, 112 °/s max
peak arm acceleration @1×	3296 °/s²
inference latency	46 ms on RTX 4090, 158 ms on CPU (i9-13900K, 8 threads)

The error tail is the expected diffusion multimodality: at decision points the policy commits to one valid behavior, scored here against the single demo chunk that happened to be recorded.

Two things to handle before running on hardware

1. Clamp the gripper to [0, 1]. The policy predicts control points, and a B-spline only lies within their convex hull — so the executed gripper command reaches 1.31 / −0.31, past the mechanical stop. Nothing downstream clamps it: _grip_downstream_to_yam is bare 1.0 - g, and the yam_server.py limiter bounds rate, not value. The plain-DP baseline does not have this issue (it predicts the trajectory directly and stays in range).

2. Start at --speed-up-times 1.0. 9% of chunks predict non-monotonic knots inside the active span (median 30 ms, max 105 ms of 1100 ms). safer_knots collapses those intervals, which is the likely source of the acceleration peaks. Acceleration scales with the square of the speed-up, so 4× implies ~53,000 °/s² at those spikes — beyond what a 100 Hz servo tracks. Setting task.dataset.relative_knots: true re-parameterizes knots as differences and is the training-side knob if you want to attack this.

Deployment

1. Environment

git clone https://github.com/B-spline-policy/bspline-policy.git
cd bspline-policy
mamba env create -f diffusion_policy/conda_environment.yaml   # creates `robodiff`
conda activate robodiff
pip install -e real_env/i2rt
pip install -e real_env/pyroki
pip install pyrealsense2          # top camera; see Cameras below
export PYTHONPATH=$PWD/bspline_policy:$PWD/diffusion_policy:$PWD/real_env/yam_teleop:$PYTHONPATH

A CPU-only torch build is fine on the robot NUC — no CUDA required for inference.

2. Get the checkpoint

hf download Dimios45/yam-duster-bspline-dp deploy_ema.ckpt --local-dir ./ckpt

3. Patch the repo for joint-space actions

Status: applied and verified in the working tree this model was trained and validated from. They are not in upstream B-spline-policy/bspline-policy — if you start from upstream, apply them yourself; the full diff is described below.

The upstream repo ships a complete YAM rollout path, but it was built for end-effector actions, because the iPhone teleop records EE poses. This model predicts joints, so the decode path does not exist there and decode_action_vector raises Unsupported action_format on a 7D joint vector. That is a difference in what was recorded, not in the file format; the LeRobot v3 container itself is fully handled by the conversion script.

Image resolution needs no change: the rollout reads dimensions from the checkpoint's own shape_meta and resizes automatically (policy_local_bspline.py:635-639). The POLICY_IMAGE_WIDTH/HEIGHT constants only feed the upstream offline converter, which is unused here.

The five changes:

1. real_env/yam_teleop/rollout_local_policy.py:11 — YAM_TELEOP_DIR points at REPO_ROOT / "simple_mobile" / "yam_teleop", which does not exist. Change to REPO_ROOT / "real_env" / "yam_teleop".

2. policy_local_utils.py::infer_action_meta — add, before the action_dim == 10 branch:

   if action_dim == 7 and "joint_pos" in obs_keys:
       action_format = "single_yam_joint"
       action_layout = "joint1..6,gripper"
   

3. policy_local_utils.py::decode_action_vector — add a matching branch. The clamp is required, not optional:

   if action_format == "single_yam_joint":
       if action_raw.size != 7:
           raise ValueError(f"Expected 7D joint action, got {action_raw.size}")
       return {"joint_pos": np.concatenate([
           action_raw[:6], np.clip(action_raw[6:7], 0.0, 1.0),
       ])}
   

4. yam_server.py::YamArm — execute_action should accept joint_pos and write self._q_cmd directly, bypassing the pyroki velocity-IK step; get_state should also return joint_pos (the self._robot.get_joint_pos() it already reads) so the observation dict matches shape_meta.

5. real_env.py / cameras.py / constants.py — add the second camera as top_image. Upstream RealEnv has a single wrist camera and only an OAKCamera class, so RealSenseCamera (pyrealsense2) and OpenCVCamera (V4L2) were added; RealEnv now raises if the top camera yields no frame. This one fails silently if skipped: policy_local_bspline.py:625-629 substitutes a black frame for any missing RGB key, so the policy runs half-blind and looks like a bad checkpoint rather than raising.

Cameras

This policy takes two views, wrist_image and top_image. They must be the same physical cameras in the same poses as during recording — a moved top camera is the likeliest cause of a model that "trained fine but does nothing sensible".

The dataset's top view is an Intel RealSense, so deploy with the RealSense path to keep the colour pipeline identical to training:

pip install pyrealsense2   # not pulled in by conda_environment.yaml

# real_env/yam_teleop/constants.py
TOP_CAMERA_TYPE = 'realsense'   # 'realsense' | 'usb' | 'oak'
TOP_CAMERA_ID   = None          # RealSense serial; None = first device found

Colour order is load-bearing. RealSenseCamera requests rs.format.rgb8 and returns frames untouched, matching how LeRobot recorded this dataset (its RealSense backend defaults to RGB and only converts to BGR on request). Verified against the frames themselves: the most chromatic pixels are channel-0 dominant — the red table marker. Do not add a cvtColor; swapping R and B is a silent domain shift that degrades the policy with no error anywhere.

Use RealSenseCamera rather than OpenCVCamera for RealSense devices — they expose several /dev/video* nodes (colour, depth, IR) and the colour index is not stable across reboots or USB ports.

Before you trust it: verify the gripper convention

The training data uses 0 = open, 1 = closed, and get_state applies YAM_GRIPPER_INVERT to convert from the i2rt raw reading. If your gripper is wired or configured differently, that inversion is backwards and the policy will open to grasp and close to release — it looks almost-working, which is the hardest failure mode here to diagnose from behaviour.

Open the gripper by hand and read it, then close it and read again:

cd real_env/yam_teleop && python -c "
from multiprocessing.managers import BaseManager as M
from constants import ARM_RPC_HOST, ARM_RPC_PORT, RPC_AUTHKEY
class Mg(M): pass
Mg.register('YamArm'); m = Mg(address=(ARM_RPC_HOST, ARM_RPC_PORT), authkey=RPC_AUTHKEY); m.connect()
print('gripper reads:', round(float(m.YamArm().get_state()['joint_pos'][6]), 3))"

Expect ~0.0 open and ~1.0 closed. If reversed, flip YAM_GRIPPER_INVERT in constants.py.

Alternative: no deploy-code changes at all

If you would rather not modify the robot control server, run forward kinematics on the recorded joints at conversion time to emit arm_pos + rotvec + gripper, producing a 10D rot6d policy identical in shape to upstream's. Then edits 2–4 disappear and the existing single_yam_rot6d decoder plus pyroki IK work untouched — only the camera edit and the path fix remain.

Costs: a retrain, an FK→IK round-trip that adds tracking error, and you must use the same URDF/TCP frame as yam_server._fk_tcp or the poses will not line up. Joint space was chosen here because the data is natively joint targets and commanding them directly skips IK entirely.

4. Bring up the arm

sudo ip link set can_follower_r up type can bitrate 1000000
python real_env/yam_teleop/yam_server.py --channel can_follower_r

5. Roll out

python real_env/yam_teleop/rollout_local_policy.py \
  --env yam --policy bspline \
  --ckpt-path ./ckpt/deploy_ema.ckpt \
  --diffusion-policy-dir $PWD/diffusion_policy \
  --control-freq 100 --data-freq 10 \
  --origin-time-scale 10 \
  --predict-before-end 0.3 \
  --speed-up-times 1.0 \
  --save --output-dir data/local_policy_rollouts

Before the first real rollout, confirm every shape_meta observation key is present and not all black — a dead camera does not raise, it just degrades the policy:

cd real_env/yam_teleop && python -c "
import torch, dill
from real_env import RealEnv
cfg = torch.load('../../ckpt/deploy_ema.ckpt', pickle_module=dill, map_location='cpu', weights_only=False)['cfg']
env = RealEnv(use_cameras=True); obs = env.get_obs()
for k in cfg.shape_meta['obs']:
    v = obs.get(k)
    print(f'  {k}:', 'MISSING' if v is None else
          (f'{v.shape} ALL BLACK' if v.ndim == 3 and not v.any() else f'{v.shape} ok'))
env.close()"

--origin-time-scale must equal the training data rate (10) — knots are stored in frame units and this converts them to seconds. --data-freq 10 matches training; --control-freq 100 matches YAM_CONTROL_HZ.

On a CPU-only NUC, raise --predict-before-end to 0.3–0.5 (roughly 2–3× measured latency) and consider --num-inference-steps 8, which nearly halves the cost. Inference runs on a daemon thread, so latency does not stall the control loop — it only has to finish before the current chunk ends. That also caps CPU-only deployment near 1–2× speed-up, since a chunk spans only ~250 ms of wall time at 4×.

Reproducing

Conversion and verification tooling: the tools/ directory of the working repo (lerobot_v3_to_robomimic.py, verify_bspline_deploy.py, strip_ckpt_for_deploy.py).

python tools/lerobot_v3_to_robomimic.py \
  --repo-id Dimios45/yam_duster_in_box \
  --output diffusion_policy/data/yam_duster_in_box.hdf5 \
  --target-fps 10 --image-size 128

cd bspline_policy && python train.py \
  --config-name=yam_duster_unet_bspline \
  hydra.run.dir=../outputs/yam_duster_bspline_full \
  training.resume=false logging.mode=offline \
  checkpoint.topk.k=601 dataloader.persistent_workers=True

Citation

@article{han2026b,
  title={B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations},
  author={Han, Xiaoshen and Xiong, Haoyu and Chen, Haonan and Liu, Chaoqi and
          Torralba, Antonio and Zhu, Yuke and Du, Yilun},
  journal={arXiv preprint arXiv:2607.09648},
  year={2026}
}