Action-conditioned latent Kalman filtering for robust spatio-temporal state estimation of deformable object

Robust spatio-temporal state estimation from noisy observations is a critical challenge in robotic manipulation. Current latent space representations derived from dynamic point clouds often suffer from severe perceptual noise and occlusions, leading to discontinuous trajectories and pronounced temporal jitter. To mitigate these artifacts, we propose an Action-Conditioned Latent Kalman Filter (AC-LKF) framework.

Leveraging a hybrid architecture of PointNet++ and Gated Recurrent Units (GRU), our method models non-linear temporal dynamics within a low-dimensional latent manifold. Instead of claiming to fully model the physical causality of deformable-object dynamics, AC-LKF incorporates the robot end-effector displacement as a low-dimensional, observable, and task-relevant kinematic prior to condition the latent state transition. Furthermore, a heteroscedastic latent filtering module adaptively estimates process and measurement uncertainties, balancing the action-conditioned prior with degraded visual inputs under noise and occlusion.

Evaluations using quantitative temporal metrics and reconstruction errors, supplemented by principal component analysis (PCA) and t-distributed stochastic neighbor embedding (t-SNE) visualizations, show that AC-LKF improves latent trajectory smoothness, information retention, and dynamic robustness by reducing abrupt observation-induced state transitions. These results indicate that kinematic action conditioning can improve the temporal consistency and robustness of learned geometric state representations for interactive deformable-object manipulation. The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

School of Mechanical and Electrical Engineering, Shenzhen University, Shenzhen, 518000, China Department of Mechanical and Materials Engineering, University of Jeddah, 22233, Jehhah, Saudi Arabia School of Electronic and Electrical Engineering, University of Leeds, Leeds, LS2 9JT, UK The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Gao, D., Melibary, M. & Hu, J. Action-conditioned latent Kalman filtering for robust spatio-temporal state estimation of deformable object. Sci Rep (2026). https://doi.org/10.1038/s41598-026-77444-2 DOI: https://doi.org/10.1038/s41598-026-77444-2