Vision based Navigation for Micro Aerial Vehicles (MAVs)

We design smart computer-vision algorithms that allow small flying robots to fly all by themselves, without any user intervention. Our flying robots only use onboard cameras and inertial sensors to "see" the world and "orientate" themselves. No GPS, no laser, no external positioning systems (e.g., Vicon), no CGI are used in these videos. Our driving motivations are search-and-rescue and remote-inspection scenarios. For example, quadrotors have the great potential to enter and rapidly explore damaged buildings in search of survivors after a catastrophe. Alternatively, they can be used in all remote-inspection operations that fall beyond the line of sight with the operator. For all these operations, it is crucial that they can navigate all by themselves.

Flightmare: A Flexible Quadrotor Simulator

Currently available quadrotor simulators have a rigid and highly-specialized structure: either are they really fast, physically accurate, or photo-realistic. In this work, we propose a paradigm-shift in the development of simulators: moving the trade-off between accuracy and speed from the developers to the end-users. We release a new modular quadrotor simulator: Flightmare. Flightmare is composed of two main components: a configurable rendering engine built on Unity and a flexible physics engine for dynamics simulation. Those two components are totally decoupled and can run independently from each other. Flightmare comes with several desirable features: (i) a large multi-modal sensor suite, including an interface to extract the 3D point-cloud of the scene; (ii) an API for reinforcement learning which can simulate hundreds of quadrotors in parallel; and (iii) an integration with a virtual-reality headset for interaction with the simulated environment. Flightmare can be used for various applications, including path-planning, reinforcement learning, visual-inertial odometry, deep learning, human-robot interaction, etc.


References

Flightmare_Yunlong

Y. Song, S. Naji, E. Kaufmann, A. Loquercio, D. Scaramuzza

Flightmare: A Flexible Quadrotor Simulator

arXiv preprint, 2020

PDF YouTube Website


Learning High-Level Policies for Model Predictive Control

High_mpc

The combination of policy search and deep neural networks holds the promise of automating a variety of decision- making tasks. Model Predictive Control (MPC) provides robust solutions to robot control tasks by making use of a dynamical model of the system and solving an optimization problem online over a short planning horizon. In this work, we leverage probabilistic decision-making approaches and the generalization capability of artificial neural networks to the powerful online optimization by learning a deep high-level policy for the MPC (High-MPC). Conditioning on robot’s local observations, the trained neural network policy is capable of adaptively selecting high-level decision variables for the low-level MPC controller, which then generates optimal control commands for the robot. First, we formulate the search of high-level decision variables for MPC as a policy search problem, specifically, a probabilistic inference problem. The problem can be solved in a closed-form solution. Second, we propose a self-supervised learning algorithm for learning a neural network high-level policy, which is useful for online hyperparameter adaptations in highly dynamic environments. We demonstrate the importance of incorporating the online adaption into autonomous robots by using the proposed method to solve a challenging control problem, where the task is to control a simulated quadrotor to fly through a swinging gate. We show that our approach can handle situations that are difficult for standard MPC.


References

ArXiv Preprint

Yunlong Song, Davide Scaramuzza

Learning High-Level Policies for Model Predictive Control

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, 2020.

PDF Code YouTube


Deep Drone Acrobatics

Performing acrobatic maneuvers with quadrotors is extremely challenging. Acrobatic flight requires high thrust and extreme angular accelerations that push the platform to its physical limits. Professional drone pilots often measure their level of mastery by flying such maneuvers in competitions. In this paper, we propose to learn a sensorimotor policy that enables an autonomous quadrotor to fly extreme acrobatic maneuvers with only onboard sensing and computation. We train the policy entirely in simulation by leveraging demonstrations from an optimal controller that has access to privileged information. We use appropriate abstractions of the visual input to enable transfer to a real quadrotor. We show that the resulting policy can be directly deployed in the physical world without any fine-tuning on real data. Our methodology has several favorable properties: it does not require a human expert to provide demonstrations, it cannot harm the physical system during training, and it can be used to learn maneuvers that are challenging even for the best human pilots. Our approach enables a physical quadrotor to fly maneuvers such as the Power Loop, the Barrel Roll, and the Matty Flip, during which it incurs accelerations of up to 3g.


References

RSS20_Kaufmann

Elia Kaufmann*, Antonio Loquercio*, René Ranftl, Matthias Müller, Vladlen Koltun, Davide Scaramuzza

Deep Drone Acrobatics

Robotics: Science and Systems (RSS), 2020.

PDF YouTube RSS2020 Pitch Video Blog Post Code


AlphaPilot: Autonomous Drone Racing

We present a novel system for autonomous, vision-based drone racing combining learned data abstraction, nonlinear filtering, and time-optimal trajectory planning. The system has successfully been deployed at the first autonomous drone racing world championship: the 2019 AlphaPilot Challenge. Contrary to traditional drone racing systems, that only detect the next gate, our approach makes use of any visible gate and takes advantage of multiple, simultaneous gate detections to compensate for drift in the state estimate and build a global map of the gates. The global map and drift-compensated state estimate allow the drone to navigate through the race course even when the gates are not immediately visible and further enable to plan a near time-optimal path through the race course in real time based on approximate drone dynamics. The proposed system has been demonstrated to successfully guide the drone through tight race courses reaching speeds up to 8m/s and has led to rank second at the 2019 AlphaPilot Challenge.


References

RSS20_Foehn

P. Foehn, D. Brescianini, E. Kaufmann, T. Cieslewski, M. Gehrig, M. Muglikar, D. Scaramuzza

AlphaPilot: Autonomous Drone Racing

Robotics: Science and Systems (RSS), 2020

PDF YouTube RSS2020 Pitch Video


Towards Low-Latency High-Bandwidth Control of Quadrotors using Event Cameras

Event cameras are a promising candidate to enable high speed vision-based control due to their low sensor latency and high temporal resolution. However, purely event-based feedback has yet to be used in the control of drones. In this work, a first step towards implementing low-latency high-bandwidth control of quadrotors using event cameras is taken. In particular, this paper addresses the problem of one-dimensional attitude tracking using a dualcopter platform equipped with an event camera. The event-based state estimation consists of a modified Hough transform algorithm combined with a Kalman filter that outputs the roll angle and angular velocity of the dualcopter relative to a horizon marked by a black-and-white disk. The estimated state is processed by a proportional-derivative attitude control law that computes the rotor thrusts required to track the desired attitude. The proposed attitude tracking scheme shows promising results of event-camera-driven closed loop control: the state estimator performs with an update rate of 1 kHz and a latency determined to be 12 ms, enabling attitude tracking at speeds of over 1600 degrees per second.


References

Towards Low-Latency High-Bandwidth Control of Quadrotors using Event Cameras

R. Sugimoto, M. Gehrig, D. Brescianini, D. Scaramuzza

Towards Low-Latency High-Bandwidth Control of Quadrotors using Event Cameras

IEEE International Conference on Robotics and Automation (ICRA), 2020

PDF YouTube ICRA2020 Video Pitch


Dynamic Obstacle Avoidance for Quadrotors with Event Cameras

Today's autonomous drones have reaction times of tens of milliseconds, which is not enough for navigating fast in complex dynamic environments. To safely avoid fast moving objects, drones need low-latency sensors and algorithms. We departed from state-of-the-art approaches by using event cameras, which are bioinspired sensors with reaction times of microseconds. Our approach exploits the temporal information contained in the event stream to distinguish between static and dynamic objects and leverages a fast strategy to generate the motor commands necessary to avoid the approaching obstacles. Standard vision algorithms cannot be applied to event cameras because the output of these sensors is not images but a stream of asynchronous events that encode per-pixel intensity changes. Our resulting algorithm has an overall latency of only 3.5 milliseconds, which is sufficient for reliable detection and avoidance of fast-moving obstacles. We demonstrate the effectiveness of our approach on an autonomous quadrotor using only onboard sensing and computation. Our drone was capable of avoiding multiple obstacles of different sizes and shapes, at relative speeds up to 10 meters/second, both indoors and outdoors.


References

Science20_Falanga

Davide Falanga, Kevin Kleber, and Davide Scaramuzza

Dynamic Obstacle Avoidance for Quadrotors with Event Cameras

Science Robotics, March 18, 2020.

PDF Supplementary Material YouTube


How Fast is Too Fast? The Role of Perception Latency in High-Speed Sense and Avoid

D. Falanga, S. Kim, D. Scaramuzza

How Fast is Too Fast? The Role of Perception Latency in High-Speed Sense and Avoid

IEEE Robotics and Automation Letters (RA-L), 2019.

PDF YouTube


Deep Drone Racing: Learning Agile Flight in Dynamic Environments




Dynamically changing environments, unreliable state estimation, and operation under severe resource constraints are fundamental challenges for robotics, which still limit the deployment of small autonomous drones. We address these challenges in the context of autonomous, vision-based drone racing in dynamic environments. A racing drone must traverse a track with possibly moving gates at high speed. We enable this functionality by combining the performance of a state-of-the-art path-planning and control system with the perceptual awareness of a convolutional neural network (CNN). The CNN directly maps raw images to a desired waypoint and speed. Given the CNN output, the planner generates a short minimum-jerk trajectory segment that is tracked by a model-based controller to actuate the drone towards the waypoint. The resulting modular system has several desirable features: (i) it can run fully on-board, (ii) it does not require globally consistent state estimation, and (iii) it is both platform and domain independent. We extensively test the precision and robustness of our system, both in simulation and on a physical platform. In both domains, our method significantly outperforms the prior state of the art. In order to understand the limits of our approach, we additionally compare against professional human drone pilots with different skill levels.


References

TRO19_Loquercio

A. Loquercio*, E. Kaufmann*, R. Ranftl, A. Dosovitskiy, V. Koltun, D. Scaramuzza

Deep Drone Racing: From Simulation to Reality with Domain Randomization

IEEE Transactions on Robotics, 2019

PDF YouTube 1 YouTube 2 Code


Deep Drone Racing

E. Kaufmann, A. Loquercio, R. Ranftl, A. Dosovitskiy, V. Koltun, D. Scaramuzza

Deep Drone Racing: Learning Agile Flight in Dynamic Environments

Conference on Robotic Learning (CoRL), Zurich, 2018.

Best Systems Paper Award!

Oral Presentation. Oral Acceptance Rate: 4.0%.

PDF YouTube


Beauty and the Beast

E. Kaufmann, M. Gehrig, P. Foehn, R. Ranftl, A. Dosovitskiy, V. Koltun, D. Scaramuzza

Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing.

This approach was used to win the IROS 2018 Autonomous Drone Race.

IEEE International Conference on Robotics and Automation (ICRA), 2019.

PDF YouTube


Information field illustration

J. Delmerico, T. Cieslewski, H. Rebecq, M. Faessler, D. Scaramuzza

Are We Ready for Autonomous Drone Racing? The UZH-FPV Drone Racing Dataset

IEEE International Conference on Robotics and Automation (ICRA), 2019.

PDF YouTube Project Webpage and Datasets Code


The Foldable Drone: A Morphing Quadrotor that can Squeeze and Fly


The recent advances in state estimation, perception, and navigation algorithms have significantly contributed to the ubiquitous use of quadrotors for inspection, mapping, and aerial imaging. To further increase the versatility of quadrotors, recent works investigated the use of an adaptive morphology, which consists of modifying the shape of the vehicle during flight to suit a specific task or environment. However, these works either increase the complexity of the platform or decrease its controllability. In this paper, we propose a novel, simpler, yet effective morphing design for quadrotors consisting of a frame with four independently rotating arms that fold around the main frame. To guarantee stable flight at all times, we exploit an optimal control strategy that adapts on the fly to the drone morphology. We demonstrate the versatility of the proposed adaptive morphology in different tasks, such as negotiation of narrow gaps, close inspection of vertical surfaces, and object grasping and transportation. The experiments are performed on an actual, fully autonomous quadrotor relying solely on onboard visual-inertial sensors and compute. No external motion tracking systems and computers are used. This is the first work showing stable flight without requiring any symmetry of the morphology.


References

RAL18_Falanga

D. Falanga, K. Kleber, S. Mintchev, D. Floreano, D. Scaramuzza

The Foldable Drone: A Morphing Quadrotor that can Squeeze and Fly

IEEE Robotics and Automation Letters (RA-L), 2018.

PDF YouTube Project page


Arxiv20_Fabris

Amedeo Fabris, Kevin Kleber, Davide Falanga, Davide Scaramuzza

Title: Geometry-aware Compensation Scheme for Morphing Drones

ArXiv Preprint 2020.

PDF YouTube


Perception Aware Navigation

Optimal control and model-based predictive control are extremely powerful methods to control systems or vehicles. However, most control approaches for MAVs use simple PID control in a cascaded structure and strictly split estimation, planning, and control into separate problems. With optimal control methods one could not only simplify the control architecture, task description, interfacing, and usability, but also take into account dynamic perception objectives and solve planning and control in one single step. Our control architectures abstract the underlying model and provide optimal control and receding horizon predictions at real-time with computation on low-power ARM processors. We also investigate the advantages of perception-aware control, where the robot's perception restrictions are taken into account in the control and planning stage and are used to improve perception performance. While such control pipelines are great for systems with known and simple dynamics, recent advances in machine learning (especially deep neural networks) have shown superior performance in very difficult high-level tasks and high-dimensional data processing. We strongly belief that both model-based and learning-based approaches should work as a union to fully exploit the advantages of both worlds. We intend to provide strong controllers as the foundation for neural-network-based high-level control and sensor data abstraction.

References

PAMPC

D. Falanga, P. Foehn, P. Lu, D. Scaramuzza

PAMPC: Perception-Aware Model Predictive Control for Quadrotors

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, 2018.

PDF YouTube Code


ICRA18_Zhang

Z. Zhang, D. Scaramuzza

Perception-aware Receding Horizon Navigation for MAVs

IEEE International Conference on Robotics and Automation, 2018.

PDF Video ICRA18 Video Pitch PPT


C. Forster, M. Pizzoli, D. Scaramuzza

Appearance-based Active, Monocular, Dense Reconstruction for Micro Aerial Vehicles

Robotics: Science and Systems, Berkely, 2014.

PDF


DroNet: Learning to Fly by Driving

Civilian drones are soon expected to be used in a wide variety of tasks, such as aerial surveillance, delivery, or monitoring of existing architectures. Nevertheless, their deployment in urban environments has so far been limited. Indeed, in unstructured and highly dynamic scenarios drones face numerous challenges to navigate autonomously in a feasible and safe way. In contrast to the traditional map-localize-plan methods, this paper explores a data-driven approach to cope with the above challenges. To do this, we propose DroNet, a convolutional neural network that can safely drive a drone through the streets of a city. Designed as a fast 8-layers residual network, DroNet produces, for each single input image, two outputs: a steering angle, to keep the drone navigating while avoiding obstacles, and a collision probability, to let the UAV recognize dangerous situations and promptly react to them. But how to collect enough data in an unstructured outdoor environment, such as a city? Clearly, having an expert pilot providing training trajectories is not an option given the large amount of data required and, above all, the risk that it involves for others vehicles or pedestrians moving in the streets. Therefore, we propose to train a UAV from data collected by cars and bicycles, which, already integrated into urban environments, would expose other cars and pedestrians to no danger. Although trained on city streets, from the viewpoint of urban vehicles, the navigation policy learned by DroNet is highly generalizable. Indeed, it allows a UAV to successfully fly at relative high altitudes, and even in indoor environments, such as parking lots and corridors.

References

3D reconstruction with an Event-based camera in real-time

A. Loquercio, A.I. Maqueda, C.R. Del Blanco, D. Scaramuzza

DroNet: Learning to Fly by Driving

IEEE Robotics and Automation Letters (RA-L), 2018.

PDF YouTube Software and Datasets


Flying in Low Light thanks to Event Cameras

In this paper, we present the first state estimation pipeline that leverages the complementary advantages of a standard camera with an event camera by fusing in a tightly-coupled manner events, standard frames, and inertial measurements. We show on the Event Camera Dataset that our hybrid pipeline leads to an accuracy improvement of 130% over event-only pipelines, and 85% over standard-frames only visual-inertial systems, while still being computationally tractable.

Furthermore, we use our pipeline to demonstrate - to the best of our knowledge - the first autonomous quadrotor flight using an event camera for state estimation, unlocking flight scenarios that were not reachable with traditional visual inertial odometry, such as low-light environments and high dynamic range scenes.


References

RAL18_VidalRebecq

T. Rosinol Vidal, H.Rebecq, T. Horstschaefer, D. Scaramuzza

Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios

IEEE Robotics and Automation Letters (RA-L), 2018.

PDF YouTube ICRA18 Video Pitch


Vision-based Autonomous Quadrotor Landing on a Moving Platform

We present a quadrotor system capable of autonomously landing on a moving platform using only onboard sensing and computing. We rely on state-of-the-art computer vision algorithms, multi-sensor fusion for localization of the robot, detection and motion estimation of the moving platform, and path planning for fully autonomous navigation. Our system does not require any external infrastructure, such as motion capture systems. No prior information about the location of the moving landing target is needed. We validate our system in both synthetic and real-world experiments using low-cost and lightweight consumer hardware. To the best of our knowledge, this is the first demonstration of a fully autonomous quadrotor system capable of landing on a moving target, using only onboard sensing and computing, without relying on any external infrastructure.


References

SSRR17_Falanga

D. Falanga, A. Zanchettin, A. Simovic, J. Delmerico, D. Scaramuzza

Vision-based Autonomous Quadrotor Landing on a Moving Platform

IEEE/RSJ International Symposium on Safety, Security and Rescue Robotics (SSRR), Shanghai, 2017.

PDF YouTube


Exploration with Multi-Rotors

Exploring and mapping previously unknown environments while avoiding collisions with obstacles is a fundamental task for autonomous robots. In scenarios where this needs to be done rapidly, multi-rotors are a good choice for the task, as they can cover ground at potentially very high velocities. Flying at high velocities, however, implies the ability to rapidly plan trajectories and to react to new information quickly. In this work, we propose an extension to classical frontier-based exploration that facilitates exploration at high speeds. The extension consists of a reactive mode in which the multi-rotor rapidly selects a goal frontier from its field of view. The goal frontier is selected in a way that minimizes the change in velocity necessary to reach it. While this approach can increase the total path length, it significantly reduces the exploration time, since the multi-rotor can fly at consistently higher speeds.


References

IROS17_Cieslewski

T. Cieslewski, E. Kaufmann, D. Scaramuzza

Rapid Exploration with Multi-Rotors: A Frontier Selection Method for High Speed Flight

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, 2017.

PDF YouTube PPT


Autonomous, Vision-based Flight and Live Dense 3D Mapping with Quadrotors


In the left video, we demonstrate our flying 3D scanner. Our quadrotor flies autonomously over an area of interest using only its onboard sensors: namely, a single, down-looking camera and an inertial measurement unit. All flight-relevant computation is performed onboard, on a smartphone processor. No pilot, GPS, or motion-capture system is required. We stream a subset of the images down to a laptop computer, where we compute a dense 3D model of the environment. Our algorithms are so efficient that we can create these models in real-time. Therefore, our system has high potential for applications in which detailed information about an unknown environment is time critical, such as during search-and-rescue missions.
In the right video, we demonstrate several capabilities of our vision-controlled quadrotors, such as autonomous way-point navigation, live, dense 3D reconstruction, and aerial-guided collaborative grasping with a ground robot. Our visual-odometry pipeline (SVO) runs in real-time onboard the quadrotor. The MAV navigates autonomously along predefinied way-points. No GPS, nor teleoperation, no motion capture system were used throughout the flight. Our live, dense 3D reconstructin algorithm runs REMODE, our REgularized MOnocular DEnse reconstruction algorithm.


We demonstrate our vision-based micro aerial vehicle on the disaster training site of the Zurich firefighters. Our visual-odometry pipeline (SVO) runs in real-time onboard the quadrotor. The MAV navigates autonomously along predefinied way-points. No GPS or teleoperation was used throughout the flight. The altitude in this experiment is 20m above ground.


In this video, our drone autonomously flyes over a person and takes 3D scan of it. Suddently, the person starts to move, trying to disturb the drone. However, our robust visual odometry and probabilistic dense reconstruction algorithms do not get disturbed by the motion of the subject and manage to get an accurate 3D scan of it and the surrounding scene. The visual odometry, planning, and control algorithms run fully onboard the drone, on a smartphone processor.


In this video, we show a vision-based drone that can autonomously execute a given trajectory and provide a live, dense 3D map of an area. This map is presented to the operator while the quadrotor is mapping, so that there are no unnecessary delays in the mission. Our system does not rely on any external positioning system (e.g., GPS or motion capture systems) as sensing, computation, and control are performed fully onboard on a smartphone processor.


References

JFR2016_Faessler

M. Faessler, F. Fontana, C. Forster, E. Mueggler, M. Pizzoli, D. Scaramuzza

Autonomous, Vision-based Flight and Live Dense 3D Mapping with a Quadrotor Micro Aerial Vehicle

Journal of Field Robotics, 2016.

PDF YouTube1 YouTube2 YouTube3 YouTube4 Software


C. Forster, M. Pizzoli, D. Scaramuzza

SVO: Fast Semi-Direct Monocular Visual Odometry

IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, 2014.

PDF YouTube Software


M. Pizzoli, C. Forster, D. Scaramuzza

REMODE: Probabilistic, Monocular Dense Reconstruction in Real Time

IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, 2014.

PDF YouTube


Continuous On-Board Monocular-Vision-based Elevation Mapping Applied to Autonomous Landing of Micro Aerial Vehicles

In this paper, we propose a resource-efficient system for real-time 3D terrain reconstruction and landing-spot detection for micro aerial vehicles. The system runs on an on-board smartphone processor and requires only the input of a single downlooking camera and an inertial measurement unit. We generate a two-dimensional elevation map that is probabilistic, of fixed size, and robot-centric, thus, always covering the area immediately underneath the robot. The elevation map is continuously updated at a rate of 1 Hz with depth maps that are triangulated from multiple views using recursive Bayesian estimation. To highlight the usefulness of the proposed mapping framework for autonomous navigation of micro aerial vehicles, we successfully demonstrate fully autonomous landing including landing-spot detection in real-world experiments.

References

ICRA2015_Forster

C. Forster, M. Faessler, F. Fontana, M. Werlberger, D. Scaramuzza

Continuous On-Board Monocular-Vision-based Elevation Mapping Applied to Autonomous Landing of Micro Aerial Vehicles

IEEE International Conference on Robotics and Automation (ICRA), Seattle, 2015.

PDF YouTube


MAV Urban Localization from Google Street View Data

We tackle the problem of globally localizing a camera-equipped micro aerial vehicle flying within urban environments for which a Google Street View image database exists. To avoid the caveats of current image-search algorithms in case of severe viewpoint changes between the query and the database images, we propose to generate virtual views of the scene, which exploit the air-ground geometry of the system. To limit the computational complexity of the algorithm, we rely on a histogram-voting scheme to select the best putative image correspondences. The proposed approach is tested on a 2km image dataset captured with a small quadroctopter flying in the streets of Zurich.

References

A.L. Majdik, D. Verda, Y. Albers-Schoenberg, D. Scaramuzza

Air-ground Matching: Appearance-based GPS-denied Urban Localization of Micro Aerial Vehicles

Journal of Field Robotics, 2015.

PDF YouTube Dataset


A. L. Majdik, D. Verda, Y. Albers-Schoenberg, D. Scaramuzza

Micro Air Vehicle Localization and Position Tracking from Textured 3D Cadastral Models

IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, 2014.

PDF


A. Majdik, Y. Albers-Schoenberg, D. Scaramuzza

MAV Urban Localization from Google Street View Data

IROS'13, IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS'13, 2013.

PDF PPT YouTube Air-ground Image Dataset


SFly: Swarm of Micro Flying Robots (EU Project: 2009-2012)

The scope of the sFly project was to develop a system of multiple vision-controlled micro aerial vehicles that are capable of autonomous navigation in GPS-denied environments, 3D mapping, and radio-beacons localization. The Sfly MAV's are fully autonomous. Flight control is done by computer vision and runs entirely onboard. No laser rangefinder or GPS are used, but just onboard cameras running at 30 Hz. An on-board monocular visual slam computes local camera poses and sparse features. These estimated poses are fused with the IMU measurements in a filter. An additional stereo system captures image data while the MAVs explore the environment. These image data are used to compute a detailed dense 3D model of the environment. An off-board visual SLAM module computes a global map by utilizing loop detection and bundle adjustment optimization.

sFly Project Website


References

D. Scaramuzza, M.C. Achtelik, L. Doitsidis, F. Fraundorfer, E.B. Kosmatopoulos, A. Martinelli, M.W. Achtelik, M. Chli, S.A. Chatzichristofis, L. Kneip, D. Gurdan, L. Heng, G.H. Lee, S. Lynen, L. Meier, M. Pollefeys, A. Renzaglia, Roland Siegwart, J.C. Stumpf, P. Tanskanen, C. Troiani, S. Weiss

Vision-Controlled Micro Flying Robots: from System Design to Autonomous Navigation and Mapping in GPS-denied Environments.

IEEE Robotics and Automation Magazine, Vol. 21, Issue 3., 2014.

PDF



This video shows automatic mapping during a controlled flight during a 200 m long flight on a village ear Zurich. The multicopter flies at more than 15 m. Concurrently to Visual SLAM, a 3D mesh is created out of the 3D points and textured is projected on to them. This 3D mesh mapping run in real-time during flight. Process happened off-board. Images were streamed to a remote 2GHz Dual-Core laptop. The algorithm run at 30 fps.


This video shows our participation at the European Micro Aerial Vehicle competition where we won the 2nd place and we were the only purely vision based autonomous multicopter. Besides being the only one using a single camera as sole sensor, we were also the the only autonomous multicopter among all the other participants! In this video the multicopter takes off from the circular spot, then uses Visual SLAM to build a map of the environment and navigate through via points specified by the user.


Only a single camera is used to control the multicopter. The blob is used to initialize the map and the scale factor at the beginning. Take-off is automatic. Feature points are tracked and used in a visual SLAM framework to build the map of the environment. The multicopter then follows autonomously the via-points created in this map and land automatically onto the second blob, which is used to show the precision of landing. The distance between the start and end position is about 3 meters. With this multicopter, we will partecipate in the EMAV (European Micro Aerial Vehicle) competition in September 2009 in Delft.

References

L. Doitsidis, S. Weiss, A. Renzaglia, M.W. Achtelik, E. Kosmatopoulos, R. Siegwart, and D. Scaramuzza. Optimal Surveillance Coverage for Teams of Micro Aerial Vehicles in GPS-denied Environments using Onboard Vision. Autonomous Robots, Volume 33, Issue 1-2, pages 173-188, 2012. [ PDF ]
S. Weiss, D. Scaramuzza, and R. Siegwart. Monocular-SLAM-Based Navigation for Autonomous Micro Helicopters in GPS-denied Environments. Journal of Field Robotics, Volume 28, issue 6, 2011. [ PDF ]
D. Eberli, D. Scaramuzza, S. Weiss, and R. Siegwart. Vision Based Position Control for MAVs Using one Single Circular Landmark. Intelligent and Robotic Systems, Springer, Volume 61, issue 1-4, January, 2011. [ PDF ]
S. Weiss, M. Achtelik, L. Kneip, D. Scaramuzza, and R. Siegwart. Intuitive 3D Maps for MAV Terrain Exploration and Obstacle Avoidance. Journal of Intelligent and Robotic Systems, Springer, Volume 61, Issue 1-4, January, 2011. [ PDF ]
G. Nuetzi, S. Weiss, D. Scaramuzza, and R. Siegwart. Fusion of IMU and Vision for Absolute Scale Estimation in Monocular SLAM. Intelligent and Robotic Systems, Springer, Volume 61, issue 1-4, January, 2011. [ PDF ]
L. Doitsidis, A. Renzaglia, S. Weiss, E. Kosmatopoulos, D. Scaramuzza, R. Siegwart. 3D Surveillance Coverage Using Maps Extracted by a Monocular SLAM Algorithm. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2011), San Francisco, September, 2011. [ PDF ]
L. Kneip, A. Martinelli, S. Weiss, D. Scaramuzza, R. Siegwart. A Closed-Form Solution for Absolute Scale Velocity Determination Combining Inertial Measurements and a Single Feature Correspondence. IEEE International Conference on Robotics and Automation (ICRA 2011), Shanghai, 2011. [ PDF ]
L. Kneip, A. Martinelli, S. Weiss, D. Scaramuzza, R. Siegwart. A Closed-Form Solution for Absolute Scale Velocity Determination Combining Inertial Measurements and a Single Feature Correspondence. IEEE International Conference on Robotics and Automation (ICRA 2011), Shanghai, 2011. [ PDF ]
L. Kneip, D. Scaramuzza, R. Siegwart. On the Initialization of Statistical Optimum Filters with Application to Motion Estimation. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2010), Taipei, October, 2010. [ PDF ]
M. Bloesch, S. Weiss, D. Scaramuzza, R. Siegwart. Vision Based MAV Navigation in Unknown and Unstructured Environments. IEEE International Conference on Robotics and Automation (ICRA 2010), Anchorage, Alaska, May, 2010. [ PDF ]
S. Zingg, D. Scaramuzza, S. Weiss, R. Siegwart. MAV Navigation through Indoor Corridors Using Optical Flow. IEEE International Conference on Robotics and Automation (ICRA 2010), Anchorage, Alaska, May, 2010. [ PDF ]