Semi-Dense 3D Reconstruction with a Stereo Event Camera



Description

Event cameras are bio-inspired sensors that offer several advantages, such as low latency, high-speed and high dynamic range, to tackle challenging scenarios in computer vision. This paper presents a solution to the problem of 3D reconstruction from data captured by a stereo event-camera rig moving in a static scene, such as in the context of stereo Simultaneous Localization and Mapping. The proposed method consists of the optimization of an energy function designed to exploit small-baseline spatio-temporal consistency of events triggered across both stereo image planes. To improve the density of the reconstruction and to reduce the uncertainty of the estimation, a probabilistic depth-fusion strategy is also developed. The resulting method has no special requirements on either the motion of the stereo event-camera rig or on prior knowledge about the scene. Experiments demonstrate our method can deal with both texture-rich scenes as well as sparse scenes, outperforming state-of-the-art stereo methods based on event data image representations.


Poster

Here is the poster used for presentation at ECCV 2018.


Citing

Please cite the following paper if you use these datasets for your research:

Semi-Dense 3D Reconstruction with a Stereo Event Camera

Y. Zhou, G. Gallego, H. Rebecq, L. Kneip, H. Li, D. Scaramuzza

Semi-Dense 3D Reconstruction with a Stereo Event Camera

European Conference on Computer Vision (ECCV), Munich, 2018.

PDF Poster YouTube


Stereo DAVIS dataset

The stereo event camera sequences used in our ECCV 2018 paper are available in the table below. The real-data sequences were recorded with two DAVIS240C sensors in stereo configuration, in a room with a motion capture system. The simulated sequence was generated using an event-based camera simulator.

We provide all event sequences as binary rosbag files for use with the Robot Operating System (ROS). The format is the one used by the RPG DVS ROS driver. The rosbag files contain events using dvs_msgs/EventArray messages. Ground truth poses are provided as geometry_msgs/PoseStamped message type.

Sequences

Scene Sequence name Download
bin rosbag (432 MB)
boxes rosbag 1 (509 MB)     rosbag 2 (950 MB)
desk rosbag 1 (525 MB)     rosbag 2 (601 MB)
monitor rosbag 1 (105 MB)     rosbag 2 (484 MB)
reader rosbag (706 MB)
simulation_3planes rosbag (65 MB). Calibration is in camera_info topic

Calibration: Intrinsic and Extrinsic parameters

This zip file (2 MB) contains the information about the calibration of the stereo DAVIS camera.
Calibration was performed using Kalibr on the grayscale frames acquired by the DAVIS camera and then it was used to undistort the events since both sensors (event-based and frame-based) share the same photoreceptor pixel array in the DAVIS camera.
We also provide a calibration rosbag (81 MB) (DAVIS frames) in case you want to calibrate it yourself.

Hand-Eye Calibration

The hand-eye calibration between the motion capture system and one of the DAVIS cameras is provided in this zip file (44MB). The file contains the input and output to our method. Please see our IJRR 2017 paper (page 6 and Fig. 4) for more details about how the hand-eye calibration method.

Camera-IMU Calibration

We do not use the IMU in our stereo method. However, we also provide a file (200 MB) to perform camera-imu calibration, and therefore enable research in stereo visual-inertial odometry with our dataset.

Additional Resources on Event Cameras: