Combining Optimal Control and Learning for Visual Navigation in Novel\n Environments

Model-based control is a popular paradigm for robot navigation because it can\nleverage a known dynamics model to efficiently plan robust robot trajectories.\nHowever, it is challenging to use model-based methods in settings where the\nenvironment is a priori unknown and can only be observed partially through\non-board sensors on the robot. In this work, we address this short-coming by\ncoupling model-based control with learning-based perception. The learning-based\nperception module produces a series of waypoints that guide the robot to the\ngoal via a collision-free path. These waypoints are used by a model-based\nplanner to generate a smooth and dynamically feasible trajectory that is\nexecuted on the physical system using feedback control. Our experiments in\nsimulated real-world cluttered environments and on an actual ground vehicle\ndemonstrate that the proposed approach can reach goal locations more reliably\nand efficiently in novel environments as compared to purely geometric\nmapping-based or end-to-end learning-based alternatives. Our approach does not\nrely on detailed explicit 3D maps of the environment, works well with low frame\nrates, and generalizes well from simulation to the real world. Videos\ndescribing our approach and experiments are available on the project website.\n