for Fast Robot Control Using Diffusion Models
Affiliation withheld for double-blind review
Diffusion models excel at generating diverse, multimodal trajectories for robotic control, yet their iterative denoising process introduces latency that can constrain real-time closed-loop execution. To address this problem, we propose a Dynamic Neural Koopman (DNK) distillation framework, which distills multistep diffusion inference into a single forward pass conditioned on observations and sampled noise. Specifically, we introduce a Factorized Dynamic Koopman (FDK) layer that models the denoising process through a latent linear transition. The noisy trajectory input is lifted into a latent space, where the FDK layer parameterizes a factorized linear operator with condition-dependent eigenvalues that adapt the noisy-to-denoised transition to the current context. We evaluate DNK across robot-control benchmarks that span locomotion, state-based, and image-based manipulation, comparing against accelerated generative-policy baselines. The results demonstrate that DNK achieves competitive or improved closed-loop performance relative to accelerated generative-policy baselines, while substantially reducing inference latency relative to the strongest high-performing one-step baselines. On long-horizon manipulation tasks, DNK achieves competitive performance with substantially fewer inference parameters than the one-step policy baselines. Hardware experiments on a physical Kinova manipulator further demonstrate millisecond-scale inference and reduced task completion time.
We report results on D4RL locomotion (Walker2d-medium-expert-v2, HalfCheetah-medium-expert-v2), Push-T (state and image), and long-horizon Franka / Relay-Kitchen manipulation.
Across these benchmarks, one-step DNK retains competitive closed-loop performance relative to accelerated generative-policy baselines while reducing inference latency to the millisecond range and using a compact student architecture.
Side-by-side rollouts for Teacher, Ours, Consistency Policy, KDM, and KDM-F on each benchmark.
Long-horizon kitchen manipulation: FDK student versus an identity-transition control (no FDK) under the same seed and protocol.
To reflect deployment-time latency, we provide synchronized replay where playback speed is adjusted by measured decision latency. In this mode, the teacher stream is slowed according to the teacher/student latency ratio, making the real-time responsiveness gap visually explicit.
On the physical Kinova Gen3, we compare the diffusion teacher and our one-step distilled policy under the same task specification, asynchronous replanning pipeline, and low-level controller (50 trials per method). Success requires reaching within 5 mm of the goal within 100 s without end-effector–obstacle collision; trials may continue past 100 s only to measure full completion time.
Kinova Gen3 hardware evaluation at 2× speed; paired clips support synchronized play and reset.
Second viewpoint for the same receding-horizon point-to-point obstacle-avoidance runs.
Aggregates from the Kinova Gen3 hardware evaluation (same task and protocol as above).
Success requires reaching within 5 mm of the goal within 100 s without collision. The teacher scores 0% (mean completion ~315 s); Ours scores 100% (~85 s). Terminal error and clearance remain comparable; mean inference latency drops from 151.00 ms to 4.08 ms.