December 18, 2024

GEM: A Generalizable Ego-Vision Multimodal World Model for Fine-Grained Ego-Motion, Object Dynamics, and Scene Composition Control

World models predict future frames from past observations and actions, making them powerful simulators for ego-vision tasks with complex dynamics, such as autonomous driving, human ego-centric activities, and drones. Nonetheless, existing world models for ego-vision have predominantly focused on the driving domain and the ego-vehicle's actions, which limits the complexity and diversity of the generated scenes. In this collaborative project between EPFL, the University of Bern, the Swiss Data Science Center, and ETH, we contribute to the development of GEM, a diffusion-based world model with a generalized control strategy. By leveraging ego-trajectories and general image features, GEM facilitates fine-grained control over ego-motion, enables control over the motion of other objects in the scene, and supports scene composition by inserting new objects. GEM is multimodal, capable of generating both videos and future depth sequences, thus providing rich semantic and spatial output contexts. While the primary focus of GEM remains in the domain of autonomous driving, we explore the adaptability of GEM to other ego-vision domain such as human activity and drone navigation. For more details, please check out our paper, project page, and code.

December 18, 2024

Multi-task Reinforcement Learning for Quadrotors

Our RAL 24 Paper introduces a multi-task reinforcement learning (MTRL) framework for quadrotor control that leverages shared dynamics and a multi-critic architecture to enable efficient knowledge transfer. Unlike single-task RL policies, our approach allows a single policy to perform diverse maneuvers, including stabilization, velocity tracking, and racing, while significantly improving sample efficiency and task performance in both simulation and real-world tests.
For more details, please check out our paper and video.

December 17, 2024

GG-SSMs: Graph-Generating State Space Models

State Space Models (SSMs) typically process data in fixed, linear sequences, limiting their ability to capture complex, non-local interactions. We introduce Graph-Generating State Space Models (GG-SSMs), which dynamically build graphs based on feature relationships, using Chazelle's MST algorithm for adaptive, efficient modeling of complex dependencies. Our method is tested on 11 datasets, including ImageNet object classification, and eye-tracking and optical flow datasets. GG-SSMs outperform existing SSMs by wide margins: achieving 84.9% top-1 accuracy on ImageNet (+1%), reducing KITTI-15 error to 2.77%, and improving eye-tracking rates by 0.33%. This dynamic, relationship-driven approach significantly boosts SSMs' representational power and efficiency.
For more details, please check out our paper.

December 12, 2024

Drift-free Visual SLAM using Digital Twins (RA-L 2024)

Our RAL24 paper introduces a novel approach that achieves accurate and globally-consistent localization by aligning the sparse 3D point cloud generated by the VIO/VSLAM system to a digital twin using point-to-plane matching; no visual data association is needed. Our method provides a 6-DoF global measurement tightly integrated into the VIO/VSLAM system. Experiments run on a high-fidelity GPS simulator and real-world data collected from a drone demonstrate that our approach outperforms state-of-the-art VIO-GPS systems and offers superior robustness against viewpoint changes compared to the state-of-the-art Visual SLAM systems. Check out our paper and code.

December 12, 2024

Student-Informed Teacher Training

To address the teacher-student asymmetry, we propose a framework for joint training of the teacher and student policies, encouraging the teacher to learn behaviors that can be imitated by the student despite the latters limited access to information and its partial observability. We motivate our method with a maze navigation task and demonstrate its effectiveness on complex vision-based quadrotor flight and manipulation tasks. For more details, please check out our paper.

December 12, 2024

Robotics meets Fluid Dynamics: A Characterization of the Induced Airflow around a Quadrotor

Our paper presents a novel, computationally efficient model for estimating the induced flow beneath quadrotors during hover. By leveraging classical turbulent flow theory and analyzing extensive flight data, we demonstrate a unified model applicable across different drone sizes. This model, requiring minimal input parameters, offers a practical solution for safer operations during inspection, mapping, agricultural applications, and many more scenarios where the predicting the strong winds generated by the drone's propeller is critical. For more details, check out our latest IEEE RA-L paper and video.

December 5, 2024

RPG Collaborates with NASA JPL to Pioneer Advanced Mars Science Helicopter

In this News Article, we talk about our collaboration with NASA JPL for Mars helicopters: Link

December 3, 2024

Actor-Critic Model Predictive Control: Differentiable Optimization meets Reinforcement Learning

Is it possible to combine the benefits of model-free reinforcement learning (RL) - known for its strong task performance and flexibility in optimizing general reward formulations - with the robustness and online replanning capabilities of model predictive control (MPC)? This extension digs deeper into the answer by studying our new framework called Actor-Critic Model Predictive Control. We conduct a deep study that exposes the benefits of the proposed approach: it achieves better out-of-distribution behaviour, better robustness to changes in the dynamics and improved sample efficiency. Additionally, we conduct an empirical analysis that reveals a relationship between the critic's learned value function and the cost function of the differentiable MPC, providing a deeper understanding of the interplay between the critic's value and the MPC cost functions. Our method achieves the same superhuman performance as state-of-the-art model-free RL, showcasing speeds of up to 21 m/s. For more details, check out our latest extension paper and video.

November 18, 2024

Our group welcomed the Unitree G1 humanoid robot's first time in Switzerland

The humanoid robot G1, developed by Unitree, was recently presented for the first time in Switzerland at the University of Zurich. Our group is proud to be part of such cutting-edge advancements in robotics, working to address the challenges of perception, planning, and control to unlock the full potential of humanoid robots. A Blick article explores the obstacles robots face in understanding and acting with common sense in daily life. Meanwhile, the NZZ highlights how real-world demonstrations reveal gaps between marketed potential and current robotic capabilities.

November 7, 2024

Empowering the blind at Cybathlon 2024

Our research on vision assistance for blind people is in today's UZH News. Read this interview of our PhD student Giovanni Cioffi to learn how our technology for flying robots was used to design the navigation algorithm of Sight Guide at Cybathlon 2024! Check the interview and the full video of the performance at Cybathlon.

November 6, 2024

New Research Assistant

We welcome Roberto Pellerito as a new Research Assistant in our lab!

November 5, 2024

Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight (CoRL 2024)

In this work, we combine the effectiveness of Reinforcement Learning (RL) and the efficiency of Imitation Learning (IL) in the context of vision-based, autonomous drone racing. We combine the task performance of RL with the sample efficiency of IL. Our experiments in both simulated and real-world environments demonstrate that our approach achieves superior performance and robustness than IL or RL alone in navigating a quadrotor through a racing course using only visual information without explicit state estimation. For more details, check out our paper, video, and website.

November 4, 2024

Monocular Event-Based Vision for Obstacle Avoidance with a Quadrotor (CoRL 2024)

We introduce a framework for simulation pre-training of learned event-vision control policies, which we hope can enable fast, agile, vision-based robots. Using this method, we present the first demonstrations of static obstacle avoidance with a quadrotor using only a monocular event camera. By leveraging depth prediction as a pretext task, we pre-train a reactive obstacle avoidance events-to-control policy with approximated events across hundreds of simulated, privileged expert rollouts, and then fine-tune a perception network with limited events-and-depth real-world data to achieve obstacle avoidance in indoor, outdoor, dark, and forest settings. For more details, please check out our paper and video.

November 1, 2024

Sight Guide: Vision Assistance for Blind People - Cybathlon 2024

Our team, Sight Guide, competed in the Vision Assistance Race at the CYBATHLON 2024 - the "cyber Olympics" designed to push the boundaries of assistive technology. In this race, our system guided a blind participant through everyday tasks such as walking along a sidewalk, sorting colors, ordering from a touchscreen, purchasing a box of tea, and navigating a forest—all powered by computer vision assistance! We used two RGB cameras and a depth camera for localization and 3D mapping and semantic understanding, and a belt for haptic feedback. Our system received the Jury Award for the most innovative and user-friendly solution. Check out our project page.

October 15, 2024

Learning Quadruped Locomotion Using Differentiable Simulation

(CoRL 2024, Oral)

While most recent advancements in legged robot control have been driven by traditional model-free reinforcement learning, we explore the potential of differentiable simulation. Differentiable simulation promises faster convergence and more stable training by computing low-variant first-order gradients using the robot model, but so far, its use for legged robot control has remained limited to simulation. This work represents the first demonstration of using differentiable simulation for controlling a real quadruped robot. Conducted in collaboration with MIT, this work has been accepted for oral presentation at CoRL 2024. For more details, please check out our paper and video.

October 14, 2024

RPG is happy to announce our new IROS 2024 paper: IronCub!

Check out our joint IROS2024 paper with the group of Daniele Pucci at the Istituto Italiano di Tecnologia where the iRonCub (the flying humanoid) learns to walk and fly using adversarial motion priors! This work presents an approach that enables automatic smooth transitions between legged and aerial locomotion. Leveraging the concept of Adversarial Motion Priors, our method allows the robot to imitate motion datasets and accomplish the desired task without the need for complex reward functions. The robot learns walking patterns from human-like gaits and aerial locomotion patterns from motions obtained using trajectory optimization. Through this process, the robot adapts the locomotion scheme based on environmental feedback using reinforcement learning, with the spontaneous emergence of mode-switching behavior.
For more details, please check out our paper and video.

October 9, 2024

Davide Scaramuzza keynote speaker at IROS 2024

Davide Scaramuzza will deliver a keynote speech at IROS 2024 AbuDhabi on Tuesday October 15th at 17:00 hrs. He will be showing how our research on agile flight led to real world impact, from the leading AR/VR headset, Meta Quest, to sustainable agriculture at SUIND, to autonomous inspection of power lines, and wildfire prevention! More info here: Link

October 7, 2024

RPG is happy to announce a new general sequence model: S7!

A key challenge in sequence modeling is efficiently handling long-range dependencies, which recent state-space models (SSMs) address but often with increased complexity. The new model, S7, improves on previous SSMs like Mamba (S6) by introducing input-dependent filtering with stable reparameterization, enabling dynamic state transitions while maintaining simplicity. Two key proofs demonstrate that S7 ensures stability in long-sequence modeling and controls gradient norms, preventing exploding or vanishing gradients. S7 consistently outperforms previous models on benchmarks such as neuromorphic datasets, Long Range Arena, and various physical and biological time series, achieving superior results with fewer inductive biases and much simpler design.
For more details, please check out our paper.

October 1st, 2024

New Lab Engineer

We welcome Yannick Armati as a new Lab Engineer in our lab!

September 17, 2024

Davide Scaramuzza distinguished speaker at ICRA40

Davide Scaramuzza is one of the distinguished speakers selected for the 40th Anniversary of the IEEE Conference on Robotics and Automation: Link

September 10, 2024

Deep Visual Odometry with Events and Frames (IROS 2024)

Our IROS24 paper presents RAMP-VO, a novel end-to-end learnable visual odometry system tailored for challenging conditions. It seamlessly integrates event-based cameras with traditional frames, utilizing Recurrent, Asynchronous, and Massively Parallel (RAMP) encoders. Despite being trained only in simulations, it outperforms both learning-based and model-based methods, demonstrating its potential for robust space navigation. For more details, check out our paper, video and code.

September 6, 2024

Structure-Invariant Range-Visual-Inertial Odometry (IROS 2024)

Can drones autonomously land on Mars without the need of a lander? This paper introduces a novel range-visual-inertial odometry system tailored for the challenges of the Mars Science Helicopter (MSH) mission, which aims to deploy the next generation of unmanned helicopters on Mars in a mid-air fashion (after atmospheric entry and descent), using a jetpack to slow down the helicopter within its control envelope. Unlike the Mars 2020 mission, which relied on a state estimation system assuming planar terrain, MSH requires a novel approach due to the complex topography of the targeted landing sites, with elevation variances of up to 8000 meters. Our system extends the state-of-the-art xVIO framework by fusing consistent range information with visual and inertial measurements, preventing metric scale drift in the absence of visual-inertial excitation (mono camera and constant velocity descent), and enabling landing on any terrain structure, without requiring any planar terrain assumption. Check out our latest IROS 2024 paper and video.

August 23, 2024

New Science Robotics paper: Wearable robots with vision technology for people with disabilities

August 14, 2024

We are hiring!

July 23, 2024

Reinforcement Learning Meets Visual Odometry (ECCV 2024)

Our ECCV24 paper tackles the challenges of VO by reframing VO as a sequential decision-making task and applying RL to adapt the VO process dynamically. Our approach introduces a learned agent within the VO pipeline, to make decisions such as keyframe and grid-size selection. Experimental results using classical VO methods demonstrate improvements in accuracy and robustness while eliminating the need for time-intensive parameter tuning. Check out our paper and code.

July 22, 2024

RSS 2024 Outstanding Demo Paper Award

We are honored to receive the RSS 2024 Outstanding Demo Paper Award for our paper "Demonstrating Agile Flight from Pixels wihtout State Stimation". Big congrats to the team! Checkout our paper, narrated video and live demo at RSS.

July 15, 2024

Kexin Shi receives UZH Semester Award

Congratulations to our former student Kexin Shi for receiving the UZH Semester Award her outstanding Master Thesis work on "Extreme Parkour with Legged Robots". Checkout the paper, video and code.

July 12, 2024

Elia Kaufmann receives European PhD Award on Systems and Control

Congratulations to our former PhD student Elia Kaufmann for receiving the European PhD Award on Systems and Control 2023 for his contribution to Learning Vision-Based Agile Flight: From Simulation to the Real World!

July 1st, 2024

Congratulations to Yunlong Song for successfully defending his Ph.D.

Congratulations to Yunlong Song for successfully defending his Ph.D. in "Learning Robot Control: From Reinforcement Learning to Differentiable Simulation"! Many thanks to the external reviewers Yuke Zhu from the U. of Texas at Austin, Marco Hutter from ETH Zurich, and Martin Riedmiller from Google Deep Mind! Yunlong's contributions are:
- To show that Reinforcement Learning (RL) outperforms Optimal Control in autonomous racing because it directly optimizes a non-differentiable task-level objective.
- To propose a policy-search-for-model-predictive-control (MPC) framework, combining RL's ability to optimize high-level task objectives with MPC's precise actuation and constraint handling.
- To introduce a differentiable simulation framework to leverage robot dynamics for more stable and efficient policy training.
- To develop a high-performance drone racing system that outperforms optimal control methods and professional pilots.
- To develop Flightmare, a flexible modular quadrotor simulator for reinforcement learning and vision-based flight.

- Video Recording of the PhD defense
- Yunlong's webpage (publications, source code, slides)
- Google Scholar

Congratulations, Yunlong!

July 1st, 2024

New Research Assistant

We welcome Ivan Alberico as a new Research Assistant in our lab!

June 20, 2024

Demonstrating Agile Flight from Pixels without State Estimation

We present the first vision-based quadrotor system that autonomously navigates through a sequence of gates at high speeds while directly mapping pixels to control commands. Like professional drone-racing pilots, our system does not use explicit state estimation and leverages the same control commands humans use (collective thrust and body rates). We demonstrate agile flight at speeds up to 40km/h with accelerations up to 2g. This is achieved by training vision-based policies with reinforcement learning (RL). The training is facilitated using an asymmetric actor-critic with access to privileged information. To overcome the computational complexity during image-based RL training, we use the inner edges of the gates as a sensor abstraction. Our approach enables autonomous agile flight with standard, off-the-shelf hardware. Check out our latest RSS 2024 paper and video.

June 18, 2024

Elia Kaufmann receives UZH Annual Award for his PhD thesis

Congratulations to our former PhD student Elia Kaufmann for receiving the prestigious Ackeret Award by the Swiss Association of Aeronautical Sciences for his PhD thesis on human-level autonomous drone flight!

June 18, 2024

Davide Scaramuzza receives IEEE Kiyo Tomiyasu Award 2024

Professor Davide Scaramuzza was honored to receive the IEEE Kiyo Tomiyasu Technical Field Award during the ICRA2024 conference 'for contributions to agile visual navigation of micro drones and low-latency robust perception with event cameras!' This is the third time this prestigious IEEE career award has gone to a roboticist since it was established over 20 years ago!
In this interview at UZH, Davide Scaramuzza talks about what this award personally means to him and the contributions that led to it.
List of all past IEEE Kiyo Tomiyasu Award recipients: link
List of all 2024 IEEE Technical Field Award recipients: link

June 17, 2024

Special Issue on Visual SLAM in the IEEE Transactions on Robotics (T-RO)

We are thrilled to announce a Special Issue on Visual SLAM in the IEEE Transactions on Robotics (T-RO). Submission deadline: December 15, 2024. We invite contributions on latest advancements on topics related but not limited to: robust SLAM, representations in SLAM (NeRF, Gaussian Splatting, etc.), semantic, object-level, and dynamic SLAM, large scale SLAM, novel datasets and benchmarks, unconventional vision sensors for SLAM (Event cameras, Thermal cameras, etc.), SLAM for robotics navigation, multi-agents SLAM, visual(-Inertial) odometry, learning-based SLAM, beyond conventional SLAM (Large Language Models, Foundation Models, etc.). Check out the Special Issue website.

June 14, 2024

MPCC++: Model Predictive Contouring Control for Time-Optimal Flight with Safety Constraints

Is it possible to fly as fast as possible while keeping safety? This paper introduces three key components that enhance the MPCC approach for drone racing. First, we provide safety guarantees in the form of a constraint and tunnel-shaped terminal set, which prevents gate collisions. Second, we augment the dynamics with a residual term that captures complex aerodynamic effects and thrust forces learned directly from real world data. Third, we use Trust Region Bayesian Optimization (TuRBO) to tune the hyperparameters of the MPC controller given a sparse reward based on lap time minimization. The proposed approach achieves similar lap times to the best state-of-the-art RL while satisfying constraints, achieving 100% success rate in simulation and real-world. Check out our latest RSS 2024 paper and video.

May 29, 2024

Low Latency Automotive Vision with Event Cameras

Check out our new paper published in Nature which addresses a critical bandwidth-latency trade-off observed in automotive vision systems based on image-based cameras. We tackle this challenge with a hybrid event + 20 Hz RGB camera sensor system, and a novel asynchronous learning method to deliver efficient, high-rate detections of traffic participants. This system achieves the same perceptual latency as a 5,000 Hz camera with the bandwidth of a 45 Hz camera without compromising accuracy, paving the way for efficient and robust perception in edge-case scenarios. For more information, read our paper and check out our code, video and dataset.

May 28, 2024

Limits of Deep Learning: Sequence Modeling through the Lens of Complexity Theory

Is Deep Learning dumb? Our proof shows SSMs (and Transformers) can't handle function composition, crucial for reasoning. Our theoretical and empirical analysis reveals these models struggle with function composition due to their architectural limitations, and even Chain-of-Thought prompting cannot resolve it (only mitigates it to some extent with exponential increase in resources) for sufficient depth (complexity) of the problem. For more information, read our paper.

May 16, 2024

Survey on Autonomous Drone Racing

We present our survey on Autonomous Drone Racing, published at the IEEE Transactions on Robotics (T-RO). The survey covers the latest developments in agile flight for both model based and learning based approaches. We include extensive coverage of drone racing competitions, simulators, open source software, and the state of the art approaches for flying autonomous drones at their limits! For more information, see our paper

May 10, 2024

Data-driven Feature Tracking for Event Cameras with and without Frames

In this work, we introduce a versatile feature tracker that can provide robust feature tracks and sparse disparity estimation while benefiting from the advantages of event cameras. Our tracker works solely with events or in a hybrid mode incorporating both events and frames. Moreover, our network architecture seamlessly extends to sparse disparity estimation in a stereo setup containing an event and a frame camera. For more details, check out our paper and code.

May 6, 2024

New Research Assistant

We welcome Maria Krinner as a new Research Assistant in our lab!

May 2, 2024

RSS24 Robust State Estimation Competition

Do you work on state estimation, Visual SLAM, or VIO? Are your algorithms robust enough for agile robotics? Take part in the RSS2024 Robust State Estimation Competition! To enter the competition, submit your trajectory results on the test sequences of the UZH FPV dataset which contains visual and inertial data recorded onboard a quadrotor flying at high speeds. This data was used to train our autonomous system that defeated the world-champion drone-racing pilots in several head-to-head races last year. The competition winner will be invited to give a talk at the workshop Towards Safe Autonomy at RSS 2024. Submission deadline: June 23rd, 2024. For more info: dataset page, competition page, and RSS workshop page.

April 27, 2024

Daniel Gehrig receives UZH Annual Award for his PhD thesis

Congratulations to our former PhD student Daniel Gehrig for receiving the prestigious UZH Annual Award for his PhD thesis "Efficient, Data-Driven Perception with Event Cameras". The thesis explores the potential of event cameras for quickly and reliably detecting objects such as traffic participants. To this end, he designed novel algorithms that are specifically tailored to these cameras and that learn based on examples. In the future, drivers can be supported and their safety increased, especially in dangerous situations.

April 22, 2024

A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception

This work proposes a hybrid model combining Spiking Neural Networks (SNN) and classical Artificial Neural Networks (ANN) to optimize power efficiency and latency in edge devices. The hybrid ANN-SNN model overcomes state transients and state decay issues while maintaining high temporal resolution, low latency, and low power consumption. In the context of 2D and 3D human pose estimation, the method achieves an 88% reduction in power consumption with only a 4% decrease in performance compared to fully ANN counterparts, and a 74% lower error compared to SNNs. For more information, have a look at our paper and code.

April 17, 2024

Event Cameras Meet SPADs for High-Speed, Low-Bandwidth Imaging

This paper proposes a novel sensor fusion combining Single Photon Avalance Diodes (SPADs) and event cameras for high-speed low-light imaging. SPAD sensors are celebrated for their single-photon sensitivity, while event cameras offer the ability to measure brightness changes at rates up to 1 MHz with low bandwidth requirements. The fusion of these technologies allows for the improved reconstruction of dynamic scenes in poor lighting conditions, achieving improvements of more than 5 dB PSNR in image quality at a high temporal resolution of 100 kHz. This is joint collaboration between RPG, Advanced Quantum Architecture lab at EPFL led by Prof. Edoardo Charbon and Camera Culture Lab at MIT led by Prof. Ramesh Raskar. For more details, check out our paper.

April 10, 2024

State Space Models for Event Cameras (CVPR 2024 Spotlight Paper)

Today, state-of-the-art deep neural networks that process event-camera data first convert a temporal window of events into dense, grid-like input representations. As such, they exhibit poor generalizability when deployed at higher inference frequencies (i.e., smaller temporal windows) than the ones they were trained on. We address this challenge by introducing state-space models (SSMs) to event-based vision. This design adapts to varying frequencies without the need to retrain the network at different frequencies. We comprehensively evaluate our approach against existing methods based on RNN and Transformer architectures across various benchmarks, including Gen1 and 1 Mpx event camera datasets. Our results demonstrate that SSM-based models train 33% faster and also exhibit minimal performance degradation when tested at higher frequencies than the training input. Have a look at our paper, code and video.

April 9, 2024

An N-Point Linear Solver for Line and Motion Estimation with Event Cameras (CVPR 2024 Oral Paper)

Our CVPR24 Oral paper presents a novel linear solver for event cameras, leveraging a suitable line parametrization to efficiently recover partial linear velocity and line parameters. Unlike existing solvers, it offers speed and numerical stability without costly root finding, handling both minimal and overdetermined systems with more than 5 events. Additionally, it introduces a velocity averaging scheme for full linear camera velocity recovery, outperforming previous methods in both stability and speed by over 600 times in extensive experiments. This is joint collaboration between RPG and the Mobile Perception Lab at ShanghaiTech led by Prof. Laurent Kneip. For more details, check out our paper.

April 2, 2024

New Postdoc and new Research Assistant

We welcome Dr. Elie Aljalbout as a new Postdoc and Johannes Heeg as a new Research Assistant in our lab!

March 28, 2024

Mitigating Motion Blur in Neural Radiance Fields with Events and Frames (CVPR 2024)

Our CVPR24 paper exploits event-based cameras to tackle the problem of reconstructing a sharp radiance field from a set of blurry images. We exploit both model-based and learning-based components. We explicitly model the blur formation process using the event double integral as an additional model-based prior. Moreover, we model the event-pixel response using an end-to-end learnable response function that enables the NeRF to diverge from the model-based solution whenever inaccurate, resulting in higher-quality reconstructions. For more details, check out our video, paper and code.

March 22, 2024

Learning Quadruped Locomotion Using Differentiable Simulation

While most recent advancements in legged robot control have been driven by traditional model-free reinforcement learning, we explore the potential of differentiable simulation. Differentiable simulation promises faster convergence and more stable training by computing low-variant first-order gradients using the robot model, but so far, its use for legged robot control has remained limited to simulation. This work represents the first demonstration of using differentiable simulation for controlling a real quadruped robot. Check out our latest arXiv paper.

March 18, 2024

Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight

In this work, we combine the effectiveness of Reinforcement Learning (RL) and the efficiency of Imitation Learning (IL) in the context of vision-based, autonomous drone racing. We combine the task performance of RL with the sample efficiency of IL. Our experiments in both simulated and real-world environments demonstrate that our approach achieves superior performance and robustness than IL or RL alone in navigating a quadrotor through a racing course using only visual information without explicit state estimation. For more details, check out our latest arXiv paper.

March 12, 2024

Davide Scaramuzza plenary speaker at ERF 2024

At the European Robotics Forum 2024, Davide Scaramuzza will share how model-based and machine learning methods, united with new, low-latency sensors are laying the foundation for better productivity and safety of future autonomous aircrafts. Click for more information: https://erf2024.eu/.

March 11, 2024

Davide Scaramuzza keynote speaker at NVIDIA GTC 2024

What will it take to fly autonomous drones as agile as human pilots? At GTC 2024, Davide Scaramuzza will share how model-based and machine learning methods, united with new, low-latency sensors are laying the foundation for better productivity and safety of future autonomous aircrafts. Click here for more information.

February 29, 2024

Contrastive Learning for Enhancing Robot Scene Transfer in Vision-based Agile Flight

We introduce an approach to learning an environment-agnostic representation to enhance the robustness of scene transfer for end-to-end vision-based agile flight. We propose an adaptive contrastive loss that dynamically adjusts the contrastive loss weight during training. The learned task-related embedding is similar across different environments and can be used to transfer the policy to unseen environments. Subsequently, we use the learned embedding to train a sensorimotor policy that takes only images and IMU as inputs and directly outputs the control commands. Our vision encoder training strategy outperforms several state-of-the-art methods in terms of task performance Check out our ICRA 2024 paper.

February 29, 2024

Actor-Critic Model Predictive Control

How can we combine the task performance and reward flexibility of model-free RL with the robustness and online replanning capabilities of MPC? We provide an answer by introducing a new framework called Actor-Critic Model Predictive Control (ACMPC). The key idea is to embed a differentiable MPC within an actor-critic RL framework. For more details, check out our latest ICRA 2024 paper, our video and our ICRA 2024 talk.

February 26, 2024

Contrastive Initial State Buffer for Reinforcement Learning (ICRA 2024)

Our ICRA24 paper introduces a Contrastive Initial State Buffer, which strategically selects states from past experiences and uses them to initialize the agent in the environment in order to guide it toward more informative states. The experiments on drone racing and legged locomotion show that our method achieves higher task performance while also speeding up training convergence. Check out our paper and code.

February 12, 2024

The UZH-FPV Dataset: New standing leaderboard

We are excited to announce that we have a new standing leaderboard for our UZH FPV Drone Racing Dataset. To enter the new leader board, participants will need to submit the results of their state estimators on the new test dataset. The new test dataset includes sequences containing visual and inertial data recorded onboard a quadrotor flying at speeds up to 100 km/h. This data was used to train our autonomous system that defeated the world champion drone racing pilots in several head-to-head races (Paper). The data is available for download here here. The new standing leaderboard is here. We are looking forward to your new participation to the UZH-FPV standing leaderboard!

February 11, 2024

Dense Continuous-Time Flow from Events and Frames (TPAMI 2024)

Our new TPAMI paper introduces a data-driven method that can estimate per-pixel continuous-time trajectories from event data and frames. If you have wondered how these two modalities can be effectively combined for motion estimation, have a look at our paper and code.

January 15, 2024

Our AI drone SWIFT in the Top 10 UZH News 2023

Our research on the first AI powered drone to defeat a human pilot made it to the top 10 UZH News 2023! Congratulations to the entire team!

January 12, 2024

AERIAL-CORE: AI-Powered Aerial Robots for Inspection and Maintenance of Electrical Power Infrastructures

We are excited to share our new paper that summarizes the results achieved in the Aerialcore project. In Aerialcore, we have collaborated with top European academic and industrial partners to automate the inspection of large, critical infrastructures, such as power lines. Large-scale infrastructures are prone to deterioration due to age, environmental influences, and heavy usage. Ensuring their safety through regular inspections and maintenance is crucial to prevent incidents that can significantly affect public safety and the environment. This paper introduces the first autonomous system that combines various innovative aerial robots. This system is designed for extended-range inspections beyond the visual line of sight, features aerial manipulators for maintenance tasks, and includes support mechanisms for human operators working at elevated heights. For more details, check out our paper and video.

January 03, 2024

Our Nature Paper in the Top Robotics Stories 2023

Our Nature paper is featured in the Top Robotics Stories of 2023 by IEEE Spectrum.

January 08, 2024

Seeing behind dynamic occlusion with event cameras

Unwanted camera occlusions, such as debris, dust, rain-drops, and snow, can severely degrade the performance of computer-vision systems. Dynamic occlusions are particularly challenging because of the continuously changing pattern. Our solution relies for the first time on the combination of a traditional camera with an event camera. When an occlusion moves across a background image, it causes intensity changes that trigger events. These events provide additional information on the relative intensity changes between foreground and background at a high temporal resolution, enabling a truer reconstruction of the background content. For more details, check out our paper.

December 13, 2023

Our startup SUIND raises 600k USD in seed round

Our startup SUIND raises 600k USD in a seed round led by Sunicon and others: Link

December 11, 2023

Revisiting Token Pruning for Object Detection and Instance Segmentation (WACV 2024)

Our latest paper on efficient Vision Transformers, accepted to WACV 2024, presents a novel token pruning method for object detection and instance segmentation, significantly accelerating inference while minimizing performance loss. Utilizing a lightweight MLP for dynamic token pruning, the method achieves up to 34% faster inference and reduces the performance drop to approximately 0.3 mAP, outperforming previous approaches on the COCO dataset. For more information, have a look at our paper, code and video.

December 5, 2023

Our lab is featured on Italian TV La7

Davide Scaramuzza talks about the research of our lab on Italian TV. First our autonomous drone racing is explained and subsequently the application of our research to search and resuce missions is presented. Watch the full video here (we are featured from 03:20 to 06:35).

November 14, 2023

Watch Davide Scaramuzza's talk at ETH Zurich

In this talk, hosted on November 2 by the Swiss Association of Aeronautical Sciences at ETH Zurich, Davide Scaramuzza presents an overview of our latest research aimed at achieving human-level performance with autonomous vision-based drones. He shows how integrating deep learning techniques with fast-response sensors, such as event cameras, enables drones to attain remarkable levels of speed and resilience, relying exclusively on onboard computation. Finally, he talks about the evolution of event cameras in academia and industry and their potential for robotics to enable low-latency, high bandwidth control.

November 12, 2023

Former Postdoc Reza Sabzevari appointed Professor at TU Delft

November 1, 2023

New Research Assistant

We welcome Christian Sprecher as a new Research Assistant in our lab!

October 23, 2023

We are hiring!

October 18, 2023

inControl Podcast features Davide Scaramuzza

Davide Scaramuzza is featured in the inControl podcast. He talks about magic, autonomous vision-based navigation, agile drone racing, and event-based cameras. The podcast is available on all common platforms, including the inControl website, Spotify, Apple Podcasts, Google Podcasts and Youtube.

October 16, 2023

IEEE Spectrum interviews Davide Scaramuzza and Adam Bry

In the recent IEEE Robotics Podcast, Evan Ackerman hosts Davide Scaramuzza and Adam Bry (CEO of Skydio) to discuss autonomous drones. They delve into how autonomy and computer vision enables super-human skills and how future drones could evolve.

October 3, 2023

Our work won the IEEE/RSJ IROS Best Paper Award

We are honored that our IEEE/RSJ IROS paper "Autonomous Power Line Inspection with Drones via Perception-Aware MPC" was selected for the Best Paper Award. Congratulations to all collaborators!

PDF Video Code

October 3, 2023

ICCV23 Oral Paper: A 5-Point Minimal Solver for Event Camera Relative Motion Estimation

We propose a novel, space-time manifold parametrization to constrain events generated by a line observed under locally constant speed. This first-of-its-kind minimal solver decodes the motion-geometry unknowns and yields a remarkable 100% success rate in linear velocity estimation on open-source datasets, surpassing existing methods. This is joint collaboration between RPG and the Mobile Perception Lab at ShanghaiTech led by Prof. Laurent Kneip. For more details and materials, check out the video, poster, and project page.

September 28, 2023

Our work selected as an IROS paper award candidate

Congratulations to Jiaxu and Giovanni whose IROS paper "Autonomous Power Line Inspection with Drones via Perception-Aware MPC" is nominated for either the conference best paper or best student paper award! Only 12 papers have been nominated out of 1,096 accepted papers: 1% nomination rate!

September 25, 2023

Our work won the Best Paper Award at IROS23 Workshop Robotic Perception and Mapping

We are happy to announce that our work "HDVIO: Improving Localization and Disturbance Estimation with Hybrid Dynamics VIO" won the best paper award at IROS23 Workshop Robotic Perception and Mapping: Frontier Vision and Learning Techniques. The paper will be presented in a spotlight talk on Thursday October 5th in Detroit. Congratulations to all collaborators! Check it out paper and video.

September 25, 2023

Our work in collaboration with ASL, ETH Zurich, won the Best Paper Award at IROS23 Workshop Robotic Perception and Mapping

We are happy to announce that our work in collaboration with ASL, ETH Zurich, "Attending Multiple Visual Tasks for Own Failure Detection" won the best paper award at IROS23 Workshop Robotic Perception and Mapping: Frontier Vision and Learning Techniques. The paper will be presented in a spotlight talk on Thursday October 5th in Detroit. Congratulations to all collaborators! Check it out the paper.

September 19, 2023

Code Release: Active Camera Exposure Control

We release the code of our camera controller that adjusts the exposure time and gain of the camera automatically. We propose an active exposure control method to improve the robustness of visual odometry in HDR (high dynamic range) environments. Our method evaluates the proper exposure time by maximizing a robust gradient-based image quality metric. Check out our paper for more details.

September 13, 2023

Reinforcement Learning vs. Optimal Control for Drone Racing

Reinforcement Learning (RL) vs. Optimal Control (OC) - why can RL achieve immpressive results beyond optimal control for many real-world robotic tasks? We investigate this question in our paper "Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning" published today in Science Robotics, available open access here!
Many works have focused on impressive results, but less attention has been paid to the systematic study of fundamental factors that have led to the success of reinforcement learning or have limited optimal control. Our results indicate that RL does not outperform OC because RL optimizes its objective better. Rather, RL outperforms OC because it optimizes a better objective: RL can directly optimize a task-level objective and can leverage domain randomization allowing the discovery of more robust control responses.
Check out our video to see our drone race autonomously with accelerations up to 12g!

September 1, 2023

AI Drone beats Human World Champions Head-to-Head Drone Race

We are thrilled to share our groundbreaking research paper published in Nature titled "Champion-Level Drone Racing using Deep Reinforcement Learning," available open access here!
We introduce "Swift," the first autonomous vision-based drone that won several fair head-to-head races against human world champions! The Swift AI drone combines deep reinforcement learning in simulation with data collected in the physical world. This marks the first time that an autonomous mobile robot has beaten human champions in a real physical sport designed for and by humans. As such it represents a milestone for mobile robotics, machine intelligence, and beyond, which may inspire the deployment of hybrid learning-based solutions in other physical systems, such as autonomous vehicles, aircraft, and personal robots, across a broad range of applications.
Curious to see "Swift" racing and know more? Check out these two videos from us and from Nature.

September 1, 2023

New PhD Student

We welcome Ismail Geles as a new PhD student in our lab!

August 30, 2023

From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection

State-of-the-art event-based deep learning methods typically convert raw events into dense input representations before they can be processed by standard networks. However, selecting this representation is very expensive, since it requires training a separate neural network for each representation and comparing the validation scores. In this work, we circumvent this bottleneck by measuring the quality of event representations with the Gromov-Wasserstein Discrepancy, which is 200 times faster to compute. This work opens a new unexplored field of explicit representation optimization. For more information, have a look at our paper. The code will be available on this link at the start of the ICCV 2023 conference.

August 25, 2023

IROS2023 Workshop: Learning Robot Super Autonomy

Do not miss our IROS2023 Workshop: Learning Robot Super Autonomy! The workshop features an incredible speakers lineup and we will have a best paper award with prize money. Checkout the agenda and join the presentations at our workshop website. Organized by Giuseppe Loianno and Davide Scaramuzza.

July 28, 2023

Our Science Robotics 2021 paper wins prestigious Chinese award!

We are truly honored to receive the prestigious Frontiers of Science Award in the category Robotics Science and Systems, which was presented on July 16th 2023 at the International Congress of Basic Science in the Beijing's People's Hall of China for our Science Robotics 2021's paper "Learning High Speed Flight in the Wild"! Congratulations to the entire team: Antonio Loquercio, Elia Kaufmann Rene Ranftl, Matthias Mueller, Vladlen Koltun. Many thanks to the award committee! Congratulations to other winners too. Paper, open-source code, and video.

July 04, 2023

Our paper on Authorship Attribution through Deep Learning accepted at PLOS ONE

We are excited to announce that our paper on authorship attribution for research papers has just been published in PLOS ONE. We developed a transformer-based AI that achieves over 70% accuracy on the newly created, largest-to-date, authorship-attribution dataset with over 2000 authors. For more information check out our PDF and open-source code.

July 03, 2023

Video Recordings of the 4th International Workshop on Event-Based Vision at CVPR 2023 available!

The recordings of the 4th international workshop on event-based vision at CVPR 2023 are available here. The event was co-organized by Guillermo Gallego, Davide Scaramuzza, Kostas Daniilidis, Cornelia Femueller, Davide Migliore.

June 13, 2023

Our CVPR Paper is Featured in Computer Vision News

Our CVPR highlight and award-candidate work "Data-driven Feature Tracking for Event Cameras" is featured on Computer Vision News. Find out more and read the complete interview with the authors Nico Messikommer, Mathias Gehrig and Carter Fang here!

Jun 13, 2023

DSEC-Detection Dataset Release

We release a new dataset for event- and frame-based object detection, DSEC-Detection based on the DSEC dataset, with aligned frames, events and object tracks. For more details visit the dataset website.

PDF YouTube Dataset Code

May 13, 2023

Training Efficient Controllers via Analytic Policy Gradient

In systems with limited compute, such as aerial vehicles, an accurate controller that is efficient at execution time is imperative. We propose an Analytic Policy Gradient (APG) method to tackle this problem. APG exploits the availability of differentiable simulators by training a controller offline with gradient descent on the tracking error. Our proposed method outperforms both model-based and model-free RL methods in terms of tracking error. Concurrently, it achieves similar performance to MPC while requiring more than an order of magnitude less computation time. Our work provides insights into the potential of APG as a promising control method for robotics.

PDF YouTube Code

May 10, 2023

We are hiring

We have multiple openings for a Scientific Research Manager, Phd students and Postdocs in Reinforcement Learning for Agile Vision-based Navigation and Computer vision with Standard Cameras and Event Cameras. Job descriptions and how to apply: https://rpg.ifi.uzh.ch/positions.html

May 04, 2023

Code Release: Tightly coupling global position measurements in VIO

We are excited to release fully open-source our code to tightly fuse global positional measurements in visual-inertial odometry (VIO)! Our code integrates global positional measurements, for example GPS, in SVO Pro, a sliding-window optimization-based VIO that uses the SVO frontend. We leverage the IMU preintegration theory to efficiently include the global position measurements in the VIO problem formulation. Our system outperforms the loosely-coupled approach in terms of absolute trajectory error up to 50% with negligible increase of the computational cost. For more information, have a look at our paper and code.

May 03, 2023

We win the ICRA Agile Movements Workshop Poster Award

Congratulations to Yunlong Song for winning the ICRA "Agile Movements: Animal Behaviour, Biomechanics, and Robot Devices" workshop poster award with his work "Fly fast with Reinforcement Learning".

April 17, 2023

Neuromorphic Optical Flow and Real-time Implementation with Event Cameras (CVPRW 2023)

We present a new spiking neural network (SNN) architecture that significantly improves optical flow prediction accuracy while reducing complexity, making it ideal for real-time applications in edge devices and robots. By leveraging event-based vision and SNNs, our solution achieves high-speed optical flow prediction with nearly two orders of magnitude less complexity, without compromising accuracy. This breakthrough paves the way for efficient real-time deployments in various computer vision pipelines. For more information, have a look at our paper.

April 11, 2023

Event-based Shape from Polarization

We introduce a novel shape-from-polarization technique using an event camera (accepted at CVPR 2023). Our setup consists of a linear polarizer rotating at high-speeds in front of an event camera. Our method uses the continuous event stream caused by the rotation to reconstruct relative intensities at multiple polarizer angles. Experiments demonstrate that our method outperforms physics-based baselines using frames, reducing the MAE by 25% in synthetic and real-world dataset. For more information, have a look at our paper.

April 07, 2023

Recurrent Vision Transformers for Object Detection with Event Cameras (CVPR 2023)

We introduce a novel efficient and highly-performant object detection backbone for event-based vision. Through extensive architecture study, we find that vision transformers can be combined with recurrent neural networks to effectively extract spatio-temporal features for object detection. Our proposed architecture can be trained from scratch on publicly available real-world data to reach state-of-the-art performance while lowering inference time compared to prior work by up to 6 times. For more information, have a look at our paper and code.

April 3, 2023

RPG and LINA Project featured in RSI

In the recent news broadcast by RSI, our lab is featured for its efforts in developing and boosting research on civil applications for drones. The LINA project at the Dübendorf airport is making its infrastructure availble to researchers and industries to facilitate the testing and developing of autonomous flying systems hardware and software. RSI [IT]

March 30, 2023

Event-based Agile Object Catching with a Quadrupedal Robot

This work the low-latency advantages of event cameras for agil object catching with a quadrupedal robot. We use the event camera to estimate the trajectory of the object, which is then caught using an RL-trained policy. Our robot catches objects at up to 15 m/s with a 83% success rate. For more information, have a look at our ICRA 2023 paper, video and open-source code.

March 10, 2023

HILTI-SLAM Challenge 2023

RPG and HILTI are organizing the ICRA2023 HILTI SLAM Challenge! Instructions here. The HILTI SLAM Challenge dataset is a real-life, multi-sensor dataset with accurate ground truth to advance the state of the art in highly accurate state estimation in challenging environments. Participants will be ranked by the completeness of their trajectories and by the achieved accuracy. HILTI is a multinational company that offers premium products and services for professionals on construction sites around the globe. Behind this vast catalog is a global team comprising of 30.000 team members from 133 different nationalities located in more than 120 countries.

March 09, 2023

LINA Testing Facility at Dübendorf Airport

UZH Magazin releases a news article about our research on autonomous drones and our new testing facility at Dübendorf Airport that enables researchers to develop autonomous systems such as drones and ground-based robots from idea to marketable product. Read the article in English or in German. More information about the LINA project can be found here.

March 7, 2023

Our Master student Fang Nan wins ETH Medal for Best Master Thesis

Fang Nan, who did his Master thesis Nonlinear MPC for Quadrotor Fault-Tolerant Control at RPG has received the ETH Medal 2023 and the Willi Studer Prize for his outstanding work. Check out his RAL 2022 paper here, which is based on his Master thesis.

January 17, 2023

Event-based Shape from Polarization

We introduce a novel shape-from-polarization technique using an event camera. Our setup consists of a linear polarizer rotating at high-speeds in front of an event camera. Our method uses the continuous event stream caused by the rotation to reconstruct relative intensities at multiple polarizer angles. Experiments demonstrate that our method outperforms physics-based baselines using frames, reducing the MAE by 25% in synthetic and real-world dataset. For more information, have a look at our paper.

January 11, 2023

Survey on Autonomous Drone Racing

We present our survey on Autonomous Drone Racing which covers the latest developments in agile flight for both model based and learning based approaches. We include extensive coverage of drone racing competitions, simulators, open source software, and the state of the art approaches for flying autonomous drones at their limits! For more information, see our paper

January 10, 2023

4th International Workshop on Event-Based Vision at CVPR 2023

The event will take place on June 19, 2023 in Vancouver, Canada. The deadline to submit a paper contribution is March 20 via CMT. More info on our website. The event is co-organized by Guillermo Gallego, Davide Scaramuzza, Kostas Daniilidis, Cornelia Femueller, Davide Migliore.

January 04, 2023

Davide Scaramuzza featured author of IEEE

We are honored that Davide Scaramuzza is featured authors on the IEEE website.

Older News