Teaching



Fall 2020 - Vision Algorithms for Mobile Robotics

UZH-DINF2039 / ETH-151-0632-00L



Goal of the Course


For a robot to be autonomous, it has to perceive and understand the world around it. This course introduces you to the key computer vision algorithms used in mobile robotics, such as image formation, filtering, feature extraction, multiple view geometry, dense reconstruction, tracking, image retrieval, event-based vision, visual-inertial odometry, Simultaneous Localization And Mapping (SLAM), and some basics of deep learning (the algorithms behind Apple ARKit, Google Visual Positioning Service, Microsoft Hololens, Magic Leap, Oculus Quest and Oculus Insight, the Mars Curiosity rover), and the Mars helicopter. Basic knowledge of algebra, geomertry, and matrix calculus are required.




Time and location


Since 05.11.2020, all lectures and exercises have been taking place on ZOOM. The links have been communicated by email.


Lectures: every Thursday from 10:15 to 12:00.


Exercises: every Thursday from 14:15 to 16:00.


Please check out the course agenda below for the exact schedule.




Instructors


Lecture Instructor: Prof. Dr. Davide Scaramuzza.

Exercise Instructors: Daniel Gehrig and Manasi Muglikar.

Office hours of Prof. Scaramuzza: Every Thursday from 16:00 to 18:00 on ZOOM (please book your slot by email by using VAMR" as email subject and he will reply with the ZOOM link)




Video Recordings


All video recordings of the lectures and exercises are available in this folder for your convenience. Alterantivey, they can be accessed from the url reported in the course program below. The password has been communicated by email to all the enrolled students.




Course Program, Slides, and Additional Reading Material


Please notice that this is a tentative schedule and that the effective content of the lecture may change from week to week.


Date

Lecture and Exercise description

Slides, exercises, and additional reading material

17.09.2020 Lecture 01 - Introduction to Computer Vision and Visual Odometry
No Exercise today.

Lecture 01 Video Recording
Lecture Slides
Visual odometry tutorial Part I
Visual odometry tutorial Part II
SLAM survey paper
24.09.2020 Lecture 02 - Image Formation: perspective projection and camera models
Exercise 01- Augmented reality wireframe cube

Lecture 02 Video Recording
Exercise 01 Video Recording
Lecture Slides (updated 18.01.2021, see changes here)
Introduction to Matlab
Exercise 01 (updated 01.10.2020)
Solutions 01 (updated 01.10.2020)
01.10.2020 Lecture 03 - Camera Calibration
Exercise 02 - PnP problem

Lecture 03 Video Recording
Exercise 02 Video Recording
Lecture Slides (updated 01.10.2020, see changes here)
Additional reading on P3P and PnP problems
Exercise 02 (updated 07.10.2020)
Solutions 02
08.10.2020 Lecture 04 - Filtering & Edge detection
No Exercise today.
Lecture 04 Video Recording
Lecture Slides
15.10.2020 Lecture 05 - Point Feature Detectors, Part 1
Exercise 03 - Harris detector + descriptor + matching

Lecture 05 Video Recording
Exercise 03 Video Recording
Lecture Slides
Exercise 03
Solution 03
22.10.2020 Lecture 06 - Point Feature Detectors, Part 2
Exercise 04 - SIFT detector + descriptor + matching

Lecture 06 Video Recording
Exercise 04 Video Recording
Lecture Slides (updated 28.10.2020, see changes here)
Additional reading on feature detection
Exercise 04
Solution 04
29.10.2020 Lecture 07 - Multiple-view geometry 1
Exercise 05 - Stereo vision: rectification, epipolar matching, disparity, triangulation

Lecture 07 Video Recording
Exercise 05 Video Recording
Lecture Slides (updated 04.01.2021, see changes here)
Additional reading on stereo image rectification
Exercise 05
Solution 05
05.11.2020 Lecture 08 - Multiple-view geometry 2
Exercise 06 - Eight-Point Algorithm

Lecture 08 Video Recording
Exercise 06 Video Recording
Lecture Slides (updated 14.11.2020, see changes here)
Additional reading on 2-view geometry
Exercise 06
Solution 06
12.11.2020 Lecture 09 - Multiple-view geometry 3
Exercise 07 - P3P algorithm and RANSAC

Lecture 09 Video Recording
Exercise 07 Video Recording
Lecture Slides (updated 12.11.2020, see changes here)
Additional reading on RANSAC
Exercise 07
Solutions 07
19.11.2020 Lecture 10 - Multiple-view geometry 4
Exercise session: Intermediate VO Integration

Lecture 10 Video Recording
Exercise VO integration - Video Recording
Lecture Slides (updated 03.01.2021, see changes here)
Additional reading on VO- accuracy evaluation
26.11.2020 Lecture 11
1st hour: Optical Flow and KLT Tracking
2nd hour: Seminar by Christian Forster, Facebook Zurich, on the Oculus Quest VIO/SLAM
1st hour of the afternoon: KLT tracking continued.
2nd hour of the afternoon: Exercise 08 - Lucas-Kanade tracker

Lecture 11 Part 1 + Facebook seminar - Video Recording
Lecture 11 Part 2 - Video Recording
Exercise 08 Video Recording
Lecture Slides (updated 26.11.2020, see changes here)
Additional reading on KLT
Exercise 08
Solutions 08
03.12.2020 Lecture 12a (1st hour) - Place recognition
Lecture 12b (2nd hour) - Dense 3D Reconstruction and Place recognition
Lecture 12c (3rd and 4th hour, replaces exercise) - Deep Learning Tutorial
Optional Exercise on Place Recogntion

Lecture 12ab - Video Recording
Lecture 12c - Video Recording
Lecture Slides on Place Recognition
Lecture Slides on Dense 3D Reconstruction
Lecture Slides on Deep Learning
Optional Exercise on Place Recogntion
Additional reading on dense 3D reconstruction
Additional reading on Bag-of-Words-based place recognition
10.12.2020 Lecture 13 - Visual inertial fusion
Exercise 09 - Bundle Adjustment

Lecture 13 - Video Recording
Exercise 09 - Video Recording
Lecture Slides (updated 18.01.2021, see changes here)
Additional reading on VIO
Exercise 09
Solutions 09
17.12.2020 Lecture 14 - Event based vision
Exercise session: Final VO Integration

Lecture 14 - Video Recording
Exercise on Exam and final VO integration - Video Recording
Lecture Slides
Additional reading on event-based vision


Date

Oral Exams

21.01.2021 to 22.01.2021 Oral exams for UZH students only.
Exam location: Online or physical in Scaramuzza's office, Andreasstrasse 15, 2.10, 8050 Zurich.
27.01.2021 to 12.02.2021 Oral exams for ETH students (the dates are handled by ETH Exam Center and are usually communicated by ETH to you via email before Christmas).
Exam location: Online or physical in Scaramuzza's office, Andreasstrasse 15, 2.10, 8050 Zurich.



Oral Exam and Exam Questions Examples (udpated 20.12.2019)


The oral exam will last 30 minutes and will consist of one application question followed by two theoretical questions. This document contains a non-exhaustive list of possible application questions and an exhaustive list of all the topics that you should learn about the course, which will be subject of discussion in the theoretical part.




Exercises


You will be required to bring your own laptop to the exercise session. You will need to have Matlab already pre-installed in your machine for the exercise.

  • ETH students: Download from here
  • UZH students: Download from here; Info on how to setup the license can be found there.
Please install all the toolboxes included in the license.




Optional Mini Project and Grading


Students taking the final exam have the opportunity to submit a mini project to increase their final grade. Notice that the mini projects are optional, that is, not mandatory. Depending on the result of the mini project (see Project Specifications from last year below (to be updated)), the student will be rewarded with a grade increase of up to 0.5 on the top of the grade of the oral exam. However, bear in mind that the mini project can be quite time consuming. Mini project specifications and files can be found in the table below. The deadline for the mini-project is Sunday, 10.01.2021, 23:59:59, and it can be submitted via e-mail to the assistants.


DescriptionLink (size)
Project Specification vo_project_statement.pdf
FAQ Frequently Asked Questions
Parking garage dataset (easy) parking.zip (208.3 MB)
KITTI 00 dataset (hard) kitti00.zip (2.3 GB)
Malaga 07 dataset (hard) malaga-urban-dataset-extract-07.zip (2.4 GB)
Matlab script to load datasets main.m


Recommended Textbooks

(All available in the NEBIS catalogue)

Unfortunately, a single textbook that covers all the material seen in class does not exist yet. However, the topics covered in class can be found in the books listed below. At the end of each lecture's slide deck, you will find a list of chapters and books where the topics can be found.

  • Computer Vision: Algorithms and Applications, by Richard Szeliski, Springer, 1st Edition, 2010. The PDF of the book can be freely downloaded from the author's webpage. Although the author is working on a 2nd edition, this is still under progress. Thus, in the lecture nodes we will reference chapters of the 1st edition.
  • Robotics, Vision and Control: Fundamental Algorithms, 2nd Ed., by Peter Corke 2017. The PDF of the book can be freely downloaded (only with ETH VPN) from the author's webpage.
  • An Invitation to 3D Vision, by Y. Ma, S. Soatto, J. Kosecka, S.S. Sastry.
  • Multiple view Geometry, by R. Hartley and A. Zisserman.
  • Chapter 4 of "Autonomous Mobile Robots", by R. Siegwart, I.R. Nourbakhsh, D. Scaramuzza. PDF




Spring 2015 - Autonomous Mobile Robots


The course is currently open to all the students of the University of Zurich and ETH (Bachelor's and Master's). Lectures take place every Monday (from 16.02.2014 to 30.05.2014) from 14:15 to 16:00 in the ETH main building (HG) in room E 1.2. Exercise take place almost every second Tuesday from 10:15 to 12:00 in the ETH main building in room G1.


The course is also given as an MOOC (Massive Open Online Course) under edX.


Course Program

Official course webpage.


Archived slides, videos, and lecture recordings can be found here.


Recommended Textbook

R. Siegwart, I.R. Nourbakhsh, and D. Scaramuzza

Introduction to autonomous mobile robots 2nd Edition (hardback)

A Bradford Book, The MIT Press, ISBN: 978-0-262-01535-6, February, 2011

The book can be bought during the first lecture or on Amazon.

MIT Website Book Website Buy