Time Lens: Event-based Video Frame Interpolation





Description

State-of-the-art frame interpolation methods generate intermediate frames by inferring object motions in the image from consecutive key-frames. In the absence of additional information, first-order approximations, i.e. optical flow, must be used, but this choice restricts the types of motions that can be modeled, leading to errors in highly dynamic scenarios. Event cameras are novel sensors that address this limitation by providing auxiliary visual information in the blind-time between frames. They asynchronously measure per-pixel brightness changes and do this with high temporal resolution and low latency. Event-based frame interpolation methods typically adopt a synthesis-based approach, where predicted frame residuals are directly applied to the key-frames. However, while these approaches can capture non-linear motions they suffer from ghosting and perform poorly in low-texture regions with few events. Thus, synthesis-based and flow-based approaches are complementary. In this work, we introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both. We extensively evaluate our method on three synthetic and two real benchmarks where we show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods. Finally, we release a new large-scale dataset in highly dynamic scenarios, aimed at pushing the limits of existing methods.

Citing

If you use this work in your research, please cite the following paper:

Time Lens: Event-based Video Frame Interpolation

S. Tulyakov*, D. Gehrig*, S. Georgoulis, J. Erbach, M. Gehrig, Y. Li, D. Scaramuzza

Time Lens: Event-based Video Frame Interpolation

IEEE Conference on Computer Vision and Pattern Recognition, 2021.

PDF Video Code Project Page and Dataset Slides


Google Colab

Use TimeLens on your own data by using our Google Colab notebook here.

Evaluation Code

Code for evaluation can be downloaded after filling out this form.

High Speed Event and RGB (HS-ERGB) dataset

The test High Speed Event and RGB (HS-ERGB) dataset used in our paper Time Lens: Event-based Video Frame Interpolation can also be downloaded after filling out this form.


Gallery

Slow Input Video Events TimeLens



















Slow Input Video Events TimeLens