Student Projects
How to apply
To apply, please send your CV, your Ms and Bs transcripts by email to all the contacts indicated below the project description. Do not apply on SiROP . Since Prof. Davide Scaramuzza is affiliated with ETH, there is no organizational overhead for ETH students. Custom projects are occasionally available. If you would like to do a project with us but could not find an advertized project that suits you, please contact Prof. Davide Scaramuzza directly to ask for a tailored project (sdavide at ifi.uzh.ch).
Upon successful completion of a project in our lab, students may also have the opportunity to get an internship at one of our numerous industrial and academic partners worldwide (e.g., NASA/JPL, University of Pennsylvania, UCLA, MIT, Stanford, ...).
Vision-based State Estimation for Flying Cars - Available
Description: Visual odometry algorithms have gone beyond academic research and are now widely used in the real world. Autonomous drones and flying cars, among many other robotic platforms, rely on VO to estimate their ego-motion. A robust VO is key to achieving full autonomy in real-world, potentially visually degraded, environments. In this project, we will study the failure points of current VO pipelines and propose solutions to improve their robustness. Our goal is to help expand the use of autonomous drones and flying cars in the real world. This project is collaboration with Volocopter, https://www.volocopter.com/
Goal: Get familiar with the state-of-the-art VO pipelines for flying vehicles. Understand the failure points of the state-of-the-art VO pipelines and propose solutions to increase robustness. We look for students with strong programming (C++ preferred), computer vision (ideally have taken Prof. Scaramuzza's class) and robotic background. Hardware experience (running code on robotic platforms) is preferred.
Contact Details: Giovanni Cioffi (cioffi@ifi.uzh.ch), Leonard Bauersfeld (bauersfeld@ifi.uzh.ch)
Thesis Type: Semester Project / Master Thesis
Exploring Multimodal Strategies for Event-Based Vision - Available
Description: The domain of Event-Based Vision, which replicates the human eye's ability to register changes within a scene, offers significant advancements in terms of power efficiency, latency, and dynamic range. This project aims to build upon these benefits by exploring the potential of a unique vision model capable of functioning across various visual modalities. The focus will be on understanding and enhancing its capabilities for cross-modal recognition and generalization in Event-Based Vision.
Goal: The primary objective of this project is to implement and optimize this unique vision model in the context of Event-Based Vision, with an aim to improve its cross-modal recognition and generalization capabilities. The end goal is to showcase the potential of this model in enhancing the functionality of Event-Based Vision systems.
Contact Details: Nikola Zubic (zubic@ifi.uzh.ch)
Thesis Type: Master Thesis
Enhancing Event Data Processing with Irregular Time Series Modeling - Available
Description: Event-based data presents unique challenges due to its irregular time intervals. Traditional techniques, such as Recurrent Neural Networks (RNNs), are frequently used for processing sequential data, but presuppose uniform time gaps between observations, which is not the case with event-based data. This project aims to address these irregularities by exploring a different approach to RNNs. This method could have potential implications for efficient processing of event data.
Goal: The ultimate objective of this project is to refine and validate an advanced approach to event processing that can effectively accommodate irregular time series data. In the course of this project, we aim to optimize the use of data in delivering high-quality results, with a particular focus on tasks traditionally considered challenging in this field. By the conclusion of the project, we anticipate having developed an advanced model that can handle the complexities of event-based data and significantly improve the efficiency of these systems.
Contact Details: Nikola Zubic (zubic@ifi.uzh.ch)
Thesis Type: Master Thesis
Bayesian Optimization for Racing Aerial Vehicle MPC Tuning - Available
Description: In recent years, model predictive control, one of the most popular methods for controlling constrained systems, has benefitted from the advancements of learning methods. Many applications showed the potential of the cross fertilization between the two fields, i.e., autonomous drone racing, autonomous car racing, etc. Most of the research efforts have been dedicated to learn and improve the model dynamics, however, the controller tuning, which has a crucial importance, have not been studied much.
Goal: The objective of this project is to implement an auto-tuning learning-based algorithm for Model Predictive Contouring Control in a racing drone setting. The controller will learn how to tune the controller weights by using specialized Bayesian optimization algorithms [1] that can explore the large dimensional controller parameters space. The learning algorithm will be first tested in simulation and then validated with hardware experiments on a racing aerial vehicle. Your project would include: - A literature research on the current state of the art about Bayesian optimization [1] for controller tuning and on MPCC literature [2] - Implementation of the state-of-the-art algorithms identified in the previous point - Development of a tailored automatic controller parameters adaptation based on Bayesian optimization - Simulation of the developed algorithms on a racing drone - Test of the algorithms on a real racing drone The thesis will be in collaboration between UZH Robotics and Perception group and ETH IDSC Intelligent Control Systems group. [1] Fröhlich, Lukas P., Melanie N. Zeilinger, and Edgar D. Klenske. "Cautious Bayesian optimization for efficient and scalable policy search." Learning for Dynamics and Control. PMLR, 2021. [2] A. Romero, S. Sun, P. Foehn, and D. Scaramuzza, “Model predictive contouring control for time-optimal quadrotor flight,” IEEE Trans. Robot., doi: 10.1109/TRO.2022.3173711.
Contact Details: Angel Romero Aguilar roagui@ifi.uzh.ch, Andrea Carron, carrona@ethz.ch, Kim Wabersich wkim@ethz.ch
Thesis Type: Master Thesis
Drone to Drone Interaction Effects - Available
Description: In a future where drones are ubiquitous, a pressing issue will be the interactions between the individual vehicles. In this project we want to study the aerodynamic effects that can be observed when multiple drones operate in close proximity. Professional human drone pilots know that racing too close to another drone can be dangerous because of the prop-wash from the leading drone. Is it possible to predict such interaction effects or use unexpected downwash effects to infer the other drones’ locations? This could, for example, be integrated in an online method (e.g. part of a VIO system) which then predicts where the other vehicle is approximately located. Requirements: - Machine learning experience (PyTorch) - Solid understanding of aerodynamics - Programming experience in C++ and Python
Goal: The goal of the project is to develop and successfully demonstrate such a method in the real world.
Contact Details: Leonard Bauersfeld ( bauersfeld AT ifi DOT uzh DOT ch), Jiaxu Xing ( jixing (at) ifi (dot) uzh (dot) ch)
Thesis Type: Master Thesis
End-to-End Vision-Based Landing - Available
Description: Landing a UAV is a critical part of every successful mission. In this project we seek to explore a learning-based system which uses information from an onboard camera (and possibly other sensors, such as an IMU) to autonomously land a multicopter on a target platform. To achieve this, the drone must recognize the defined target and then fly towards it to land. In comparison to the state-based approaches which uses the targets’ geometry to infer a relative position estimate, the envisioned approach should directly reason based on images (or some other high-level feature representation like semantic segmentation) to robustly land the vehicle. Requirements: - Machine learning experience (PyTorch) - Computer Vision experience (ideally our Vision-Algorithm for mobile robotics) - Programming experience in C++ and Python
Goal: The goal of the project is to develop and successfully demonstrate such a method in the real world.
Contact Details: Leonard Bauersfeld ( bauersfeld AT ifi DOT uzh DOT ch), Jiaxu Xing ( jixing (at) ifi (dot) uzh (dot) ch)
Thesis Type: Master Thesis
Localization techniques for drone racing - Available
Description: For fast and agile flight, most approaches require precise knowledge of the metric state. In contrast to a classical SLAM setting, drone racing offers additional features. In this project, we want to evaluate and compare different strategies for localization in this drone-racing scenario. The following classes of methods could be investigated: - classic feature-based SLAM - learned features with classic SLAM pipeline - learning-based localization - filtering based approaches Requirements: - Machine learning experience (TensorFlow and/or PyTorch) - Programming experience in C++ and Python
Goal: The goal of the project is to gain a detailed understanding of which method is best suited for - real-time localization and - offline postprocessing of the data.
Contact Details: Leonard Bauersfeld (bauersfeld AT ifi DOT uzh DOT ch), Giovanni Cioffi ( cioffi (at) ifi (dot) uzh (dot) ch)
Thesis Type: Master Thesis
Computational Photography and Videography - Available
Description: Computational Photography is a hot topic in computer vision because it finds widespread applications in mobile devices. Traditionally, the problem has been studied using frames from a single camera. Today, mobile devices feature multiple cameras and sensors that can be combined to push the frontier in computational photography and videography. In previous work (https://youtu.be/eomALySSGVU), we have successfully reconstructed high-speed, HDR video from events. In this project, we aim for combining information from a standard and event camera to exploit their complementary nature. Applications range from high-speed, HDR video to deblurring and beyond. Contact us for more details.
Contact Details: Mathias Gehrig (mgehrig at ifi.uzh.ch); Daniel Gehrig (dgehrig at ifi.uzh.ch)
Thesis Type: Master Thesis
Deep learning based motion estimation from events - Available
Description: Optical flow estimation is the mainstay of dynamic scene understanding in robotics and computer vision. It finds application in SLAM, dynamic obstacle detection, computational photography, and beyond. However, extracting the optical flow from frames is hard due to the discrete nature of frame-based acquisition. Instead, events from an event camera indirectly provide information about optical flow in continuous time. Hence, the intuition is that event cameras are the ideal sensors for optical flow estimation. In this project, you will dig deep into optical flow estimation from events. We will make use of recent innovations in neural network architectures and insights of event camera models to push the state-of-the-art in the field. Contact us for more details.
Goal: The goal of this project is to develop a deep learning based method for dense optical flow estimation from events. Strong background in computer vision and machine learning required.
Contact Details: Mathias Gehrig, mgehrig (at) ifi (dot) uzh (dot) ch
Thesis Type: Semester Project / Master Thesis
3D reconstruction with event cameras - Available
Description: Event cameras are bio-inspired sensors that offer several advantages, such as low latency, high-speed and high dynamic range, to tackle challenging scenarios in computer vision. Research on structure from motion and multi-view stereo with images has produced many compelling results, in particular accurate camera tracking and sparse reconstruction. Active sensors with standard cameras like Kinect have been used for dense scene reconstructions. Accurate and efficient reconstructions using event-camera setups is still an unexplored topic. This project will focus on solving the problem of 3D reconstruction using active perception with event cameras .
Goal: The goal is to develop a system for accurate mapping of complex and arbitrary scenes using depth acquired by an event camera setup. We seek a highly motivated student with the following minimum qualifications: - Excellent coding skills in Python and C++ - At least one course in computer vision (multiple view geometry) - Strong work ethic - Excellent communication and teamwork skills Preferred qualifications: - Experience with machine learning Contact for more details.
Contact Details: Manasi Muglikar, muglikar (at) ifi (dot) uzh (dot) ch
Thesis Type: Semester Project / Master Thesis
Event-based depth estimation - Available
Description: Depth estimation plays an important role in many computer vision and robotics applications, such as augmented reality, navigation, or industrial inspection. Structured light (SL) systems estimate depth by actively projecting a known pattern on the scene and observing with a camera how light interacts (i.e., deforms and reflects) with the surfaces of the objects. This project will focus on event-based depth estimation using structured light systems. The resulting approach would make structured light systems suitable for generating high-speed scans.
Goal: The goal is to develop a system for 3D depth maps with event cameras. Preferred candidate should have knowledge of computer vision and strong programming skills in Python, C++.
Contact Details: Manasi Muglikar, muglikar (at) ifi (dot) uzh (dot) ch
Thesis Type: Semester Project / Master Thesis
Learning to calibrate an event camera - Available
Description: Camera calibration is an important prerequisite for 3D computer vision tasks. Calibration techniques currently used for event cameras require a special calibration target with blinking pattern. This project focuses on developing a toolkit to calibrate an event camera using deep learning methods. The project will build on state of the art deep learning techniques for events and evaluate on camera calibration task.
Goal: The goal of this project is to develop and evaluate deep learning tools for event camera for the task of calibration.
Contact Details: Manasi Muglikar, muglikar (at) ifi (dot) uzh (dot) ch , Mathias Gehrig, mgehrig (at) ifi (dot) uzh (dot) ch
Thesis Type: Semester Project / Bachelor Thesis / Master Thesis
Model-based Reinforcement Learning for Autonomous Drone Racing - Available
Goal: The objective of this project is to modify an existing model-free Reinforcement Learning pipeline for drones to a model-based Reinforcement Learning pipeline. The goal is to investigate potential performance improvements of the reinforcement learning algorithm by incorporating a model of the drone's dynamics, which will allow the algorithm to make more informed decisions. This will result in faster learning and better generalization, leading to better performance in real-world scenarios. To accomplish this goal, the student will need to research and implement various model-based reinforcement learning algorithms and evaluate their performance in a simulation environment for drone navigation. The student will also need to fine-tune the parameters of the algorithm to achieve optimal performance. The final product will be a pipeline that can be used to train a drone to navigate in a variety of environments with improved efficiency and accuracy. Applicants should have a strong background in both model-free and model-based reinforcement learning techniques, programming in C++ and Python, and a good understanding of nonlinear dynamic systems. Additional experience in signal processing, machine learning, as well as being comfortable operating in a hands-on environment is highly desired.
Contact Details: Please send your CV and transcripts (bachelor and master), and any projects you have worked on that you find interesting to Angel Romero (roagui AT ifi DOT uzh DOT ch) and Yunlong Song (song AT ifi DOT uzh DOT ch)
Thesis Type: Semester Project / Master Thesis
Development of Off-Board Vision System Software for Autonomous Drones - Available
Goal: Design and develop the software for real-time image transmission from the drone camera to the receiver. Implement and test the software to ensure its functionality and performance. Evaluate the system's performance, specially latency, under different conditions. Demonstrate closed-loop capabilities of the platform by using an offboard Visual Odometry pipeline. The project will require the use of image processing libraries, wireless communication protocols, and programming languages such as Python and C++.
Contact Details: Please send your CV and transcripts (bachelor and master), and any projects you have worked on that you find interesting to Angel Romero (roagui AT ifi DOT uzh DOT ch) and Alex Barden (barden AT ifi DOT uzh DOT ch)
Thesis Type: Semester Project / Master Thesis
Data-driven Event Generation from Images - Available
Description: Many frame-based datasets exist with labels created by tedious manual labor. In contrast, only a few event-based datasets are available since event cameras are relatively new sensors. Thus, in the scope of this project, we want to leverage frame-based datasets by generating synthetic events based on frames. Compared to existing work, we want to generate synthetic events in a data-driven manner to reduce the simulation-to-reality gap.
Goal: In this project, the student applies current state-of-the-art deep learning concepts for image generation to create artificial events from standard frames. Since multiple state-of-the-art deep learning methods will be explored in the scope of this project, a good background in deep learning is required. If you are interested, we are happy to provide more details.
Contact Details: Nico Messikommer [nmessi (at) ifi (dot) uzh (dot) ch], Mathias Gehrig [mgehrig (at) ifi (dot) uzh (dot) ch]
Thesis Type: Semester Project / Master Thesis
Data-driven Keypoint Extractor for Event Data - Available
Description: Neuromorphic cameras exhibit several amazing properties such as robustness to HDR scenes, high-temporal resolution, and low power consumption. Thanks to these characteristics, event cameras are applied for camera pose estimation for fast motions in challenging scenes. A common technique for camera pose estimation is the extraction and tracking of keypoints on the camera plane. In the case of event cameras, most existing keypoint extraction methods are handcrafted manually. As a new promising direction, this project tackles the keypoint extraction in a data-driven fashion based on recent advances in frame-based keypoint extractors.
Goal: The project aims to develop a data-driven keypoint extractor, which computes interest points in event data. Based on a previous student project (submitted to CVPR23), the approach will leverage neural network architectures to extract and describe keypoints in an event stream. The student should have prior programming experience in a deep learning framework and completed at least one course in computer vision.
Contact Details: Nico Messikommer [nmessi (at) ifi (dot) uzh (dot) ch], Mathias Gehrig [mgehrig (at) ifi (dot) uzh (dot) ch]
Thesis Type: Semester Project / Master Thesis
Domain Transfer between Events and Frames - Available
Description: During the last years, a vast collection of frame-based datasets was collected for countless tasks. In comparison, event-based datasets represent only a tiny fraction of the available datasets. Thus, it is highly promising to use labelled frame datasets to train event-based networks as current data-driven approaches heavily rely on labelled data.
Goal: In this project, the student extends upon a previous student project (published at ECCV22) and current advances from the UDA literature in order to transfer multiple tasks from frames to events. The approach should be validated on several tasks (segmentation, object detection, etc.) in challenging environments (night, high-dynamic scenes) to highlight the benefits of event cameras. As several deep learning methods are used as tools for the task transfer, a strong background in deep learning is required. If you are interested, we are happy to provide more details.
Contact Details: Nico Messikommer [nmessi (at) ifi (dot) uzh (dot) ch], Mathias Gehrig [mgehrig (at) ifi (dot) uzh (dot) ch]
Thesis Type: Semester Project / Master Thesis
Efficient Learning-aided Visual Inertial Odometry - Available
Description: Recent works have shown that deep learning (DL) techniques are beneficial for visual inertial odometry (VIO). Different ways to include DL in VIO have been proposed: end-to-end learning from images to poses, replacing one/more block/-s of a standard VIO pipeline with learning-based solutions, and include learning in a model-based VIO block. The project will start with a study of the current literature on learning-based VIO/SLAM algorithms and an evaluation of how/where/when DL is beneficial for VIO/SLAM. We will use the results of this evaluation to enhance a current state-of-the-art VIO pipeline with DL, focusing our attention on algorithm efficiency at inference time. The developed learning-aided VIO pipeline will be compared to existing state-of-the-art model-based algorithms, with focus on robustness, and deployed on embedded platforms (Nvidia Jetson TX2 or Xavier).
Goal: Enhance standard VIO algorithms with DL techniques to improve robustness. Benchmark the proposed algorithm against existing state-of-the-art standard VIO algorithms. Deploy the proposed algorithm on embedded platforms. We look for students with strong computer vision background and familiar with common software tools used in DL (for example, PyTorch or TensorFlow).
Contact Details: Giovanni Cioffi (cioffi@ifi.uzh.ch), Manasi Muglikar (muglikar@ifi.uzh.ch)
Thesis Type: Master Thesis
Developing Smart Vision Assistive Technology - Available
Description: More than 200 million people are estimated to have moderate or severe vision impairment in 2020. Their lack of autonomy limits the completion of many daily living activities. In this project, we will focus on applying robotics techniques, such as state estimation and path planning, to help visually impaired people to navigate unknown and unstructured environments. Image credits: Katzschmann et al. 2018.
Goal: This project aims to develop a navigation solution for visually impaired people. The student will work on state estimation, such as VIO and SLAM, and path planning algorithms from the robotic domain and make them suitable for the navigation of visually impaired people. A successful navigation system will be deployed during the 2024 edition of the Cybathlon Competition, https://cybathlon.ethz.ch/en. The project will be carried out at RPG in collaboration with the Autonomous System Lab, ETH Zurich.
Contact Details: Giovanni Cioffi (cioffi@ifi.uzh.ch)
Thesis Type: Semester Project / Master Thesis
Adversarial Robustness in Event-Based Neural Networks - Available
Description: The robustness and reliability of neural networks are of utmost importance in several computer vision applications, especially in automotive applications where real-time predictions are crucial for safe and efficient operation. In this context, event-based cameras, due to their unique property of capturing changes in the scene, have shown impressive performance in low-latency prediction tasks such as object detection, tracking, and optical flow prediction. However, in order to be widely adopted in the real world, the robustness and reliability of such event-based networks have to be properly studied and verified. Until now, however, these aspects have been overlooked in the event-based literature. We look for students with strong programming (Pyhton/Matlab) and computer vision background. Additionally, knowledge in machine learning frameworks (pytorch, tensorflow) is required.
Goal: The project will focus on studying various neural network architectures for event-based inference datasets and evaluate their performance in the presence of adversarial attacks.
Contact Details: Marco Cannici (cannici AT ifi DOT uzh DOT ch)
Thesis Type: Semester Project / Master Thesis
Neural-based scene reconstruction and synthesis using event cameras - Available
Description: Purely learning-based methods leveraging implicit scene representations have shown impressive results in the reconstruction and synthesis of complex scenes from just a few images, largely surpassing those of traditional methods such as Structure-from-motion, photogrammetry, and image-based rendering. Due to their recent introduction, their advantages over traditional methods are still being explored in the field of computer vision. In particular, their use in conjunction with event-based cameras, bio-inspired sensors with improved latency, temporal resolution, and dynamic range, is still under-explored.
Goal: The project will focus on exploring the use of event-based cameras in neural-based scene reconstruction and synthesis, extending available approaches to event-based data. We look for students with strong programming (Pyhton/Matlab) and computer vision background. Additionally, knowledge of machine learning frameworks (pytorch, tensorflow) is required.
Contact Details: Marco Cannici (cannici AT ifi DOT uzh DOT ch)
Thesis Type: Semester Project / Master Thesis
Efficient Processing of Event Data for Deep Learning - Available
Description: Event Cameras show great potential for high-speed and low-latency computer vision and robotics applications. In recent years, we have particularly focused on designing novel data-driven approaches to maximize performance and minimize latency. Still, efficient and effective feature extraction from event data remains challenging. This project investigates novel approaches proposed in the machine-learning and signal-processing community to maximize the benefits of event cameras in time-critical scenarios. Applications of this project range include optical flow prediction, object detection, and tracking to closed-loop control with event-based vision in the loop. Contact us for more details.
Goal: This project aims to develop a new differentiable module for efficient and effective low-level event data processing. To achieve this goal, we will investigate new techniques recently proposed in the machine learning and signal processing literature. We will integrate the proposed method with state-of-the-art deep learning models to enable low-latency inference in applications ranging from optical flow prediction, object detection/tracking, and closed-loop drone navigation with vision in the loop.
Contact Details: Mathias Gehrig (mgehrig at ifi.uzh.ch)
Thesis Type: Master Thesis