Vision based Navigation for Micro Aerial Vehicles (MAVs)

We design smart computer-vision algorithms that allow small flying robots to fly all by themselves, without any user intervention. Our flying robots only use onboard cameras and inertial sensors to "see" the world and "orientate" themselves. No GPS, no laser, no external positioning systems (e.g., Vicon), no CGI are used in these videos. Our driving motivations are search-and-rescue and remote-inspection scenarios. For example, quadrotors have the great potential to enter and rapidly explore damaged buildings in search of survivors after a catastrophe. Alternatively, they can be used in all remote-inspection operations that fall beyond the line of sight with the operator. For all these operations, it is crucial that they can navigate all by themselves.

Autonomous, Vision-based Flight and Live Dense 3D Mapping with Quadrotors


In the left video, we demonstrate our flying 3D scanner. Our quadrotor flies autonomously over an area of interest using only its onboard sensors: namely, a single, down-looking camera and an inertial measurement unit. All flight-relevant computation is performed onboard, on a smartphone processor. No pilot, GPS, or motion-capture system is required. We stream a subset of the images down to a laptop computer, where we compute a dense 3D model of the environment. Our algorithms are so efficient that we can create these models in real-time. Therefore, our system has high potential for applications in which detailed information about an unknown environment is time critical, such as during search-and-rescue missions.
In the right video, we demonstrate several capabilities of our vision-controlled quadrotors, such as autonomous way-point navigation, live, dense 3D reconstruction, and aerial-guided collaborative grasping with a ground robot. Our visual-odometry pipeline (SVO) runs in real-time onboard the quadrotor. The MAV navigates autonomously along predefinied way-points. No GPS, nor teleoperation, no motion capture system were used throughout the flight. Our live, dense 3D reconstructin algorithm runs REMODE, our REgularized MOnocular DEnse reconstruction algorithm.


We demonstrate our vision-based micro aerial vehicle on the disaster training site of the Zurich firefighters. Our visual-odometry pipeline (SVO) runs in real-time onboard the quadrotor. The MAV navigates autonomously along predefinied way-points. No GPS or teleoperation was used throughout the flight. The altitude in this experiment is 20m above ground.


In this video, our drone autonomously flyes over a person and takes 3D scan of it. Suddently, the person starts to move, trying to disturb the drone. However, our robust visual odometry and probabilistic dense reconstruction algorithms do not get disturbed by the motion of the subject and manage to get an accurate 3D scan of it and the surrounding scene. The visual odometry, planning, and control algorithms run fully onboard the drone, on a smartphone processor.


In this video, we show a vision-based drone that can autonomously execute a given trajectory and provide a live, dense 3D map of an area. This map is presented to the operator while the quadrotor is mapping, so that there are no unnecessary delays in the mission. Our system does not rely on any external positioning system (e.g., GPS or motion capture systems) as sensing, computation, and control are performed fully onboard on a smartphone processor.


References

JFR2016_Faessler

M. Faessler, F. Fontana, C. Forster, E. Mueggler, M. Pizzoli, D. Scaramuzza

Autonomous, Vision-based Flight and Live Dense 3D Mapping with a Quadrotor Micro Aerial Vehicle

Journal of Field Robotics, 2016.

PDF YouTube1 YouTube2 YouTube3 YouTube4 Software


C. Forster, M. Pizzoli, D. Scaramuzza

SVO: Fast Semi-Direct Monocular Visual Odometry

IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, 2014.

PDF YouTube Software


M. Pizzoli, C. Forster, D. Scaramuzza

REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time

IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, 2014.

PDF YouTube


Appearance-based Active Vision for Micro Aerial Vehicles

We investigate the following problem: given the image of a scene, what is the trajectory that a robotmounted camera should follow to allow optimal dense 3D reconstruction? The solution we propose is based on maximizing the information gain over a set of candidate trajectories. In order to estimate the information that we expect from a camera pose, we introduce a novel formulation of the measurement uncertainty that accounts for the scene appearance (i.e., texture in the reference view), the scene depth, and the vehicle pose. We successfully demonstrate our approach in the case of realtime, monocular reconstruction from a micro aerial vehicle and validate the effectiveness of our solution in both synthetic and real experiments. To the best of our knowledge, this is the first work on active, monocular dense reconstruction, which chooses motion trajectories that minimize perceptual ambiguities inferred by the texture in the scene.

References

C. Forster, M. Pizzoli, D. Scaramuzza

Appearance-based Active, Monocular, Dense Reconstruction for Micro Aerial Vehicles

Robotics: Science and Systems, Berkely, 2014.

PDF


Continuous On-Board Monocular-Vision-based Elevation Mapping Applied to Autonomous Landing of Micro Aerial Vehicles

In this paper, we propose a resource-efficient system for real-time 3D terrain reconstruction and landing-spot detection for micro aerial vehicles. The system runs on an on-board smartphone processor and requires only the input of a single downlooking camera and an inertial measurement unit. We generate a two-dimensional elevation map that is probabilistic, of fixed size, and robot-centric, thus, always covering the area immediately underneath the robot. The elevation map is continuously updated at a rate of 1 Hz with depth maps that are triangulated from multiple views using recursive Bayesian estimation. To highlight the usefulness of the proposed mapping framework for autonomous navigation of micro aerial vehicles, we successfully demonstrate fully autonomous landing including landing-spot detection in real-world experiments.

References

ICRA2015_Forster

C. Forster, M. Faessler, F. Fontana, M. Werlberger, D. Scaramuzza

Continuous On-Board Monocular-Vision-based Elevation Mapping Applied to Autonomous Landing of Micro Aerial Vehicles

IEEE International Conference on Robotics and Automation (ICRA), Seattle, 2015.

PDF YouTube


MAV Urban Localization from Google Street View Data

We tackle the problem of globally localizing a camera-equipped micro aerial vehicle flying within urban environments for which a Google Street View image database exists. To avoid the caveats of current image-search algorithms in case of severe viewpoint changes between the query and the database images, we propose to generate virtual views of the scene, which exploit the air-ground geometry of the system. To limit the computational complexity of the algorithm, we rely on a histogram-voting scheme to select the best putative image correspondences. The proposed approach is tested on a 2km image dataset captured with a small quadroctopter flying in the streets of Zurich.

References

A.L. Majdik, D. Verda, Y. Albers-Schoenberg, D. Scaramuzza

Air-ground Matching: Appearance-based GPS-denied Urban Localization of Micro Aerial Vehicles

Journal of Field Robotics, 2015.

PDF YouTube Dataset


A. L. Majdik, D. Verda, Y. Albers-Schoenberg, D. Scaramuzza

Micro Air Vehicle Localization and Position Tracking from Textured 3D Cadastral Models

IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, 2014.

PDF


A. Majdik, Y. Albers-Schoenberg, D. Scaramuzza

MAV Urban Localization from Google Street View Data

IROS'13, IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS'13, 2013.

PDF PPT YouTube Air-ground Image Dataset


SFly: Swarm of Micro Flying Robots (EU Project: 2009-2012)

The scope of the sFly project was to develop a system of multiple vision-controlled micro aerial vehicles that are capable of autonomous navigation in GPS-denied environments, 3D mapping, and radio-beacons localization. The Sfly MAV's are fully autonomous. Flight control is done by computer vision and runs entirely onboard. No laser rangefinder or GPS are used, but just onboard cameras running at 30 Hz. An on-board monocular visual slam computes local camera poses and sparse features. These estimated poses are fused with the IMU measurements in a filter. "An additional stereo system captures image data while the MAVs explore the environment. These image data are used to compute a detailed dense 3D model of the environment. An off-board visual SLAM module computes a global map by utilizing loop detection and bundle adjustment optimization.

sFly Project Website


References

D. Scaramuzza, M.C. Achtelik, L. Doitsidis, F. Fraundorfer, E.B. Kosmatopoulos, A. Martinelli, M.W. Achtelik, M. Chli, S.A. Chatzichristofis, L. Kneip, D. Gurdan, L. Heng, G.H. Lee, S. Lynen, L. Meier, M. Pollefeys, A. Renzaglia, Roland Siegwart, J.C. Stumpf, P. Tanskanen, C. Troiani, S. Weiss

Vision-Controlled Micro Flying Robots: from System Design to Autonomous Navigation and Mapping in GPS-denied Environments.

IEEE Robotics and Automation Magazine, Vol. 21, Issue 3., 2014.

PDF



This video shows automatic mapping during a controlled flight during a 200 m long flight on a village ear Zurich. The helicopter flies at more than 15 m. Concurrently to Visual SLAM, a 3D mesh is created out of the 3D points and textured is projected on to them. This 3D mesh mapping run in real-time during flight. Process happened off-board. Images were streamed to a remote 2GHz Dual-Core laptop. The algorithm run at 30 fps.


This video shows our participation at the European Micro Aerial Vehicle competition where we won the 2nd place and we were the only purely vision based autonomous helicopter. Besides being the only one using a single camera as sole sensor, we were also the the only autonomous helicopter among all the other participants! In this video the helicopter takes off from the circular spot, then uses Visual SLAM to build a map of the environment and navigate through via points specified by the user.


Only a single camera is used to control the helicopter. The blob is used to initialize the map and the scale factor at the beginning. Take-off is automatic. Feature points are tracked and used in a visual SLAM framework to build the map of the environment. The helicopter then follows autonomously the via-points created in this map and land automatically onto the second blob, which is used to show the precision of landing. The distance between the start and end position is about 3 meters. With this helicopter, we will partecipate in the EMAV (European Micro Aerial Vehicle) competition in September 2009 in Delft.

References

L. Doitsidis, S. Weiss, A. Renzaglia, M.W. Achtelik, E. Kosmatopoulos, R. Siegwart, and D. Scaramuzza. Optimal Surveillance Coverage for Teams of Micro Aerial Vehicles in GPS-denied Environments using Onboard Vision. Autonomous Robots, Volume 33, Issue 1-2, pages 173-188, 2012. [ PDF ]
S. Weiss, D. Scaramuzza, and R. Siegwart. Monocular-SLAM-Based Navigation for Autonomous Micro Helicopters in GPS-denied Environments. Journal of Field Robotics, Volume 28, issue 6, 2011. [ PDF ]
D. Eberli, D. Scaramuzza, S. Weiss, and R. Siegwart. Vision Based Position Control for MAVs Using one Single Circular Landmark. Intelligent and Robotic Systems, Springer, Volume 61, issue 1-4, January, 2011. [ PDF ]
S. Weiss, M. Achtelik, L. Kneip, D. Scaramuzza, and R. Siegwart. Intuitive 3D Maps for MAV Terrain Exploration and Obstacle Avoidance. Journal of Intelligent and Robotic Systems, Springer, Volume 61, Issue 1-4, January, 2011. [ PDF ]
G. Nuetzi, S. Weiss, D. Scaramuzza, and R. Siegwart. Fusion of IMU and Vision for Absolute Scale Estimation in Monocular SLAM. Intelligent and Robotic Systems, Springer, Volume 61, issue 1-4, January, 2011. [ PDF ]
L. Doitsidis, A. Renzaglia, S. Weiss, E. Kosmatopoulos, D. Scaramuzza, R. Siegwart. 3D Surveillance Coverage Using Maps Extracted by a Monocular SLAM Algorithm. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2011), San Francisco, September, 2011. [ PDF ]
L. Kneip, A. Martinelli, S. Weiss, D. Scaramuzza, R. Siegwart. A Closed-Form Solution for Absolute Scale Velocity Determination Combining Inertial Measurements and a Single Feature Correspondence. IEEE International Conference on Robotics and Automation (ICRA 2011), Shanghai, 2011. [ PDF ]
L. Kneip, A. Martinelli, S. Weiss, D. Scaramuzza, R. Siegwart. A Closed-Form Solution for Absolute Scale Velocity Determination Combining Inertial Measurements and a Single Feature Correspondence. IEEE International Conference on Robotics and Automation (ICRA 2011), Shanghai, 2011. [ PDF ]
L. Kneip, D. Scaramuzza, R. Siegwart. On the Initialization of Statistical Optimum Filters with Application to Motion Estimation. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2010), Taipei, October, 2010. [ PDF ]
M. Bloesch, S. Weiss, D. Scaramuzza, R. Siegwart. Vision Based MAV Navigation in Unknown and Unstructured Environments. IEEE International Conference on Robotics and Automation (ICRA 2010), Anchorage, Alaska, May, 2010. [ PDF ]
S. Zingg, D. Scaramuzza, S. Weiss, R. Siegwart. MAV Navigation through Indoor Corridors Using Optical Flow. IEEE International Conference on Robotics and Automation (ICRA 2010), Anchorage, Alaska, May, 2010. [ PDF ]