for Fast Robot Control Using Diffusion Models
Affiliation withheld for double-blind review
Diffusion models excel at generating diverse, multimodal trajectories for robotic control, yet their iterative denoising process introduces latency that can constrain real-time closed-loop execution. To address this problem, we propose a Dynamic Neural Koopman (DNK) distillation framework, which distills multistep diffusion inference into a single forward pass conditioned on observations and sampled noise. Specifically, we introduce a Factorized Dynamic Koopman (FDK) layer that models the denoising process through a latent linear transition. The noisy trajectory input is lifted into a latent space, where the FDK layer parameterizes a factorized linear operator with condition-dependent eigenvalues that adapt the noisy-to-denoised transition to the current context. We evaluate DNK across robot-control benchmarks that span locomotion, state-based, and image-based manipulation, comparing against accelerated generative-policy baselines. The results demonstrate that DNK achieves competitive or improved closed-loop performance relative to accelerated generative-policy baselines, while substantially reducing inference latency relative to the strongest high-performing one-step baselines. On long-horizon manipulation tasks, DNK achieves competitive performance with substantially fewer inference parameters than the one-step policy baselines. Hardware experiments on a physical Kinova manipulator further demonstrate millisecond-scale inference and reduced task completion time.
We report results on D4RL locomotion (Walker2d-medium-expert-v2, HalfCheetah-medium-expert-v2), Push-T (state and image), and long-horizon Franka / Relay-Kitchen manipulation.
Across these benchmarks, one-step DNK retains competitive closed-loop performance relative to accelerated generative-policy baselines while reducing inference latency to the millisecond range and using a compact student architecture.
Side-by-side rollouts for Teacher, Ours, Consistency Policy, KDM, and KDM-F on each benchmark.
Long-horizon kitchen manipulation: FDK student versus an identity-transition control (no FDK) under the same seed and protocol.
To reflect deployment-time latency, we provide synchronized replay where playback speed is adjusted by measured decision latency. In this mode, the teacher stream is slowed according to the teacher/student latency ratio, making the real-time responsiveness gap visually explicit.
On the physical Kinova Gen3, we compare the diffusion teacher and our one-step distilled policy under the same task specification, asynchronous replanning pipeline, and low-level controller (50 trials per method). The goal tolerance is 5 mm without end-effector–obstacle collision; we report completion time, inference latency, and clearance (no deadline-based success rate).
Kinova Gen3 hardware evaluation; paired clips support synchronized play and reset.
Second viewpoint for the same receding-horizon point-to-point obstacle-avoidance runs.
Aggregates from the Kinova Gen3 hardware evaluation (same task and protocol as above). Deadline-based success rate is not reported.
Mean completion time is ~315 s for the teacher and ~85 s for Ours. Terminal error and clearance remain comparable; mean inference latency drops from 151.00 ms to 4.08 ms.