High Speed and High Dynamic Range Video
with an Event Camera

Training Dataset

The training dataset used in this work can be downloaded here (44.7 GB).


Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous events instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images.

In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams.

Our quantitative experiments show that our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. As an additional contribution, we demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data. We release the reconstruction code and a pre-trained model to enable further research.

Reconstruction Code

The reconstruction code and a pretrained model are available here.

High Speed and HDR Datasets

The event camera sequences used in our paper High Speed and High Dynamic Range Video with an Event Camera are available in the table below. Each sequence was recorded with two sensors: (i) a 640x480 event camera (Samsung DVS Gen3), and (ii) a conventional RGB camera (Huawei P20 Pro).

We thank Cedric Scheerlinck, Matthias Faessler, and his family for valuable help in recording these datasets.


Please cite the following paper if you use these datasets for your research:

High Speed and High Dynamic Range Video with an Event Camera

H. Rebecq, R. Ranftl, V. Koltun, D. Scaramuzza

High Speed and High Dynamic Range Video with an Event Camera

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.

PDF YouTube Code Project Page


Events (Samsung DVS Gen3) Reference Video (Huawei P20)
gun_bullet_gnome.zip (12.8 MB) gun_bullet_gnome.mp4 (8.7 MB)
gun_bullet_mug.zip (14.1 MB) gun_bullet_mug.mp4 (9.0 MB)
popping_water_balloon.zip (43.4 MB) popping_water_balloon.mp4 (4.3 MB)
hdr_selfie.zip (141 MB) hdr_selfie.mp4 (111.4 MB)
hdr_tunnel.zip (236 MB) hdr_tunnel.mp4 (26.2 MB)
popping_air_balloon popping_air_balloon.zip (4.8 MB) popping_air_balloon.mp4 (9.0 MB)
hdr_sun hdr_sun.zip (80.1 MB) hdr_sun.mp4 (17.4 MB)

Driving Event Camera Datasets

The following driving datasets consist of a number of sequences that were recorded with a VGA (640x480) event camera (Samsung DVS Gen3) and a conventional RGB camera (Huawei P20 Pro) placed on the windshield of a car driving through Zurich.

We provide all event sequences as binary rosbag files for use with the Robot Operating System (ROS). The format is the one used by the RPG DVS ROS driver. The rosbag files contain events using dvs_msgs/EventArray messages.

We thank Cedric Scheerlinck for his valuable help in recording these datasets.

Events (Samsung DVS Gen3) Reference Video (Huawei P20)
highway1.bag (5.7 GB) highway1.mp4 (227.7 MB)
highway2.bag (5.0 GB) highway2.mp4 (162.4 MB)
highway3.bag (4.5 GB) highway3.mp4 (161 MB)
parking1.bag (3.6 GB) parking1.mp4 (78.1 MB)
parking2.bag (4.0 GB) parking2.mp4 (91.5 MB)
uzh_parking1.bag (1.8 GB) uzh_parking1.mp4 (109 MB)
street1.bag (2.7 GB) street1.mp4 (52.9 MB)
street2.bag (1.4 GB) street2.mp4 (38.8 MB)
street3.bag (1.6 GB) street3.mp4 (57.0 MB)
street4.bag (3.0 GB) street4.mp4 (159 MB)
street5.bag (7.1 GB) street5.mp4 (200 MB)
back1.bag (4.9 GB) back1.mp4 (133 MB)
back2.bag (1.8 GB) back2.mp4 (84.1 MB)
back3.bag (3.6 GB) back3.mp4 (99.6 MB)
back4.bag (2.8 GB) back4.mp4 (72.5 MB)
back6.bag (615 Mb) back6.mp4 (31.1 MB)
back7.bag (2.0 GB) back7.mp4 (111.6 MB)
back8.bag (1.8 GB) back8.mp4 (51 MB)
back9.bag (9.8 GB) back9.mp4 (145.1 MB)
back10.bag (1.7 GB) back10.mp4 (56.6 MB)
back11 back11.bag (481 Mb) NA
back12.bag (1.2 GB) back12.mp4 (45.7 MB)

Driving datasets (with sun in the field-of-view)

Driving sequences recorded early in the morning (when the sun is still low in the sky), with the car facing the sun.

Events (Samsung DVS Gen3) Reference Video (Huawei P20)
sun1.bag (6.1 GB) sun1.mp4 (157 MB)
sun2.bag (3.5 GB) sun2.mp4 (115 MB)
sun3.bag (2.9 GB) sun3.mp4 (127 MB)
sun4.bag (3.2 GB) sun4.mp4 (131 MB)
sun5.bag (1.3 GB) sun5.mp4 (67.4 MB)
sun6.bag (3.7 GB) sun6.mp4 (109 MB)
sun7.bag (212 Mb) sun7.mp4 (47.8 MB)
sun8.bag (4.6 GB) sun8.mp4 (126 MB)
sun9.bag (2.2 GB) sun9.mp4 (60.9 MB)
sun10.bag (2.6 GB) sun10.mp4 (110 MB)
sun11.bag (3.9 GB) sun11.mp4 (102 MB)
sun12 sun12.bag (4.7 GB) NA
sun13.bag (4.3 GB) sun13.mp4 (99.6 MB)
sun14.bag (1.9 GB) sun14.mp4 (68.2 MB)
sun15.bag (2.3 GB) sun15.mp4 (75.9 MB)
sun16.bag (3.9 GB) sun16.mp4 (77.6 MB)
sun17.bag (2.5 GB) sun17.mp4 (64.4 MB)