Illusory Control with Instant Virtual World Environment

We proposed a teleoperation method, illusory control (IC), that provides a comfortable operation experience using a seamless transition between real and pre-prepared virtual environments. Therefore, the mobile robot with IC could function solely in familiar environments. To make IC applicable in unfamiliar environments, this study proposes a novel method, instant IC, that eliminates the requirement for a pre-prepared virtual environment. The proposed robot system can instantly generate a virtual environment using actual 360° images of the robot in motion, utilizing instant neural graphics primitives and neural radiance fields. The 360° images allow the entire surrounding environment to be virtualized without requiring specific camera orientations. In addition, by optimizing the density of neural radiance fields using depth estimation results beforehand, the reconstruction accuracy at unknown poses can be guaranteed. Furthermore, we propose a depth scaling method based on the actual measurements obtained by LiDAR to increase the consistency of virtual and real environments. With this instant virtual environment, the proposed system enables teleoperation in unknown environments via the seamless transition between real and virtual environments. The experimental results exhibit consistent and smooth back-and-forth transitions between virtual and real space in mobile robot teleoperation.