Teaching



Fall 2021 - Vision Algorithms for Mobile Robotics

UZH-DINF2039 / ETH-151-0632-00L



Registration for ETH students


This couse is a core course of the ETH Master in Robotics, Systems and Control. Differently from the past years, from this year there will be a single registration for all students via UZH. To do so, you will first need to register to get a UZH account. Afterwards, you can login and book the course. All the instructions can be found here.

ETH students should mind the following deadlines: the deadline to register for a UZH account is September 17 at 23:59hrs Zurich time. Once you receive the login credentials (it may take a few days), you can finally book the course until October 15 23:59.




Goal of the Course


For a robot to be autonomous, it has to perceive and understand the world around it. This course introduces you to the key computer vision algorithms used in mobile robotics, such as image formation, filtering, feature extraction, multiple view geometry, dense reconstruction, tracking, image retrieval, event-based vision, visual-inertial odometry, Simultaneous Localization And Mapping (SLAM), and some basics of deep learning (the algorithms behind Apple ARKit, Google Visual Positioning Service, Microsoft Hololens, Magic Leap, Oculus Quest and Oculus Insight, the Mars Perceverance rover), and the Mars helicopter. Basic knowledge of algebra, geomertry, and matrix calculus (product, inversion, eigenvalue decomposition) are required.




Time and location


All lectures and exercises will be taking place on site in the UZH City campus (next to UZH and ETH main buildings) in SOC-1-106 (located on floor F) and will also be video recorded. If the front door of the building is locked (it happened to us last week), please use the side door (located on the right side of the building).


Lectures: every Thursday from 8:00 to 9:45 am. Please notice that the lecture starts at 8 o'clock and not 8:15!


Exercises: every Thursday from 12:15 to 13:45.


Please check out the course agenda below for the exact schedule.




Covid Certificates Required for Physical Attendance


As per UZH decision of September 10th, 2021, all students and teaching staff are required to have a covid certificate. Masks will no longer be mandatory in classrooms due to the certificate requirement. However, UZH continues to strongly recommend that all attendees continue to wear masks in classrooms. More info here.




Instructors


Lecture Instructor: Prof. Dr. Davide Scaramuzza.

Exercise Instructors: Manasi Muglikar and Nico Messikommer.

Office hours of Prof. Scaramuzza: Every Thursday from 16:00 to 18:00 on ZOOM or in person (please book your slot by email by using VAMR" as email subject).




Video Recordings and Student Forum


All video recordings of the lectures and exercises will be made available within the course online platform OLAT. Please use the Forum therein to ask questions to the instructors.




Course Program, Slides, and Additional Reading Material


Please notice that this is a tentative schedule and that the effective content of the lecture may change from week to week.


Date

Lecture and Exercise description

Slides, exercises, and additional reading material

23.09.2021 Lecture 01 - Introduction to Computer Vision and Visual Odometry
No Exercise today.
Lecture Slides
Additional reading on Visual Odometry
30.09.2021 Lecture 02 - Image Formation: perspective projection and camera models
Exercise 01- Augmented reality wireframe cube
Lecture Slides
Matlab primer
Matlab primer code
Linear Algebra Primer from Stanford University
Linear Algebra interactive tool
Exercise 01
Solutions 01
07.10.2021 Lecture 03 - Camera Calibration
Exercise 02 - PnP problem
Lecture Slides (last updated on Oct. 14th, see changes)
Additional reading on P3P and PnP problems
Exercise 02
Solutions 02
14.10.2021 Lecture 04 - Filtering & Edge detection
No Exercise today.
Lecture Slides
21.10.2021 Lecture 05 - Point Feature Detectors, Part 1
Exercise 03 - Harris detector + descriptor + matching
Lecture Slides (last updated on Oct. 21st, see changes)
Exercise 03
Solutions 03
28.10.2021 Lecture 06 - Point Feature Detectors, Part 2
Exercise 04 - SIFT detector + descriptor + matching
Lecture Slides (last updated on Nov. 3rd, see changes)
Additional reading on feature detection
Exercise 04
Solutions 04
04.11.2021 Lecture 07 - Multiple-view geometry 1
Exercise 05 - Stereo vision: rectification, epipolar matching, disparity, triangulation
Lecture Slides
Additional reading on stereo image rectification
Exercise 05
Solutions 05
11.11.2021 Lecture 08 - Multiple-view geometry 2
Exercise 06 - Eight-Point Algorithm
Lecture Slides
Additional reading on 2-view geometry
Exercise 06
Solutions 06
18.11.2021 Lecture 09 - Multiple-view geometry 3
Exercise 07 - P3P algorithm and RANSAC
Lecture Slides
Additional reading on RANSAC
Exercise 07
Solutions 07
25.11.2021 Lecture 10 - Multiple-view geometry 4
Exercise session: Intermediate VO Integration
Lecture Slides (updated 02.12.2021, see changes here)
Additional reading on open-source VO algorithms
02.12.2021 Lecture 11
1st hour: seminar by Dr. Jeff Delaune from NASA-JPL: "Vision-Based Navigation for Mars Helicopters"
2nd hour: Optical Flow and KLT Tracking
Exercise 08 - Lucas-Kanade tracker
Lecture Slides
Additional reading on KLT
NASA presentation
Additional reading on NASA/JPL talk
Exercise 08
Solutions 08
09.12.2021 Lecture 12a (1st hour) - Place recognition
Lecture 12b (2nd hour) - Dense 3D Reconstruction
Lecture 12c (3rd and 4th hour, replaces exercise) - Deep Learning Tutorial
Optional Exercise on Place Recogntion
Lecture Slides on Place Recognition
Lecture Slides on Dense 3D Reconstruction
Lecture Slides on Deep Learning
Optional Exercise on Place Recogntion
Additional reading on dense 3D reconstruction
Additional reading on Bag-of-Words-based place recognition
16.12.2021 Lecture 13 - Visual inertial fusion
Exercise 09 - Bundle Adjustment
Lecture Slides
Additional reading on VIO
Exercise 09
Solutions 09
Exercise slides
23.12.2021 Lecture 14 - Event based vision
Exercise session: Final VO Integration
Lecture Slides
Additional reading on event-based vision
13.01.2022 Written exam: from 08:00 to 10:00 am. More details will be given durnig the semester.



Exam


The written exam will take place on January 13th, 2022 from 08:00 to 10:00 am. More details will be given durnig the semester.




Exercises


You will be required to bring your own laptop to the exercise session. You will need to have Matlab already pre-installed in your machine for the exercise.

  • ETH students: Download from here
  • UZH students: Download from here; Info on how to setup the license can be found there.
Please install all the toolboxes included in the license.




Optional Mini Project and Grading


Students taking the final exam have the opportunity to submit a mini project to increase their final grade. Notice that the mini projects are optional, that is, not mandatory. Depending on the result of the mini project, the student will be rewarded with a grade increase of up to 0.5 on the top of the grade of the oral exam. However, bear in mind that the mini project can be quite time consuming. Students, who want to submit a mini project, should work in a group of minimum 2 and max 4 persons. Mini project specifications and files can be found in the table below. The deadline for the mini-project is Sunday, 09.01.2022, 23:59:59, and it can be submitted via e-mail to the assistants.


DescriptionLink (size)
Project Specification vo_project_statement.pdf
FAQ Frequently Asked Questions
Parking garage dataset (easy) parking.zip (208.3 MB)
KITTI 05 dataset (hard) kitti05.zip (1.4 GB)
Malaga 07 dataset (hard) malaga-urban-dataset-extract-07.zip (4.4 GB)
Matlab script to load datasets main.m


Recommended Textbooks

(All available in the Swisscovery catalogue)

Unfortunately, a single textbook that covers all the material seen in class does not exist yet. However, the topics covered in class can be found in the books listed below. At the end of each lecture's slide deck, you will find a list of chapters and books where the topics can be found.

  • Computer Vision: Algorithms and Applications, by Richard Szeliski, Springer, 2nd Edition, 2021. The PDF of the book can be freely downloaded from the author's webpage. Although the author is working on a 2nd edition, this is still under progress. Thus, in the lecture nodes we will reference chapters of the 1st edition.
  • Chapter 4 of "Autonomous Mobile Robots", by R. Siegwart, I.R. Nourbakhsh, D. Scaramuzza. PDF
  • Robotics, Vision and Control: Fundamental Algorithms, 2nd Ed., by Peter Corke 2017. The PDF of the book can be freely downloaded (only with ETH VPN) from the author's webpage.
  • An Invitation to 3D Vision, by Y. Ma, S. Soatto, J. Kosecka, S.S. Sastry.
  • Multiple view Geometry, by R. Hartley and A. Zisserman.




Spring 2007-2015 - Autonomous Mobile Robots


From 2007 to 2015 I was co-lecturer of the course Autonomous Mobile Robots together with Prof. Roland Siegwart. The old Power Point slides and videos can be found here. The lectures of the course inspired the second edition of the omonym book Autonomous Mobile Robots below.

The course continues to be taught by Prof. Siegwart at the following link.

The course is also available as an MOOC (Massive Open Online Course) on the popular online platform edX.



Reference book

R. Siegwart, I.R. Nourbakhsh, and D. Scaramuzza

Introduction to autonomous mobile robots 2nd Edition (hardback)

A Bradford Book, The MIT Press, ISBN: 978-0-262-01535-6, February, 2011

The book can be bought during the first lecture or on Amazon.

MIT Website Buy