Video frame interpolation for high dynamic range sequences captured with dual-exposure sensors

Eurographics 2023


Uğur Çoğalan1     Mojtaba Bemana1     Hans-Peter Seidel1     Karol Myszkowski1    

1 MPI Informatik

Abstract

Video frame interpolation (VFI) enables many important applications such as slow motion playback and frame rate conversion. However, one major challenge in using VFI is accurately handling high dynamic range (HDR) scenes with complex motion. To this end, we explore the possible advantages of dual-exposure sensors that readily provide sharp short and blurry long exposures that are spatially registered and whose ends are temporally aligned. This way, motion blur registers temporally continuous information on the scene motion that, combined with the sharp reference, enables more precise motion sampling within a single camera shot. We demonstrate that this facilitates a more complex motion reconstruction in the VFI task, as well as HDR frame reconstruction that so far has been considered only for the originally captured frames, not in-between interpolated frames. We design a neural network trained in these tasks that clearly outperforms existing solutions. We also propose a metric for scene motion complexity that provides important insights into the performance of VFI methods at test time.

Citation

Uğur Çoğalan, Mojtaba Bemana, Hans-Peter Seidel, Karol Myszkowski
Video frame interpolation for high dynamic range sequences captured with dual-exposure sensors
Computer Graphics Forum (Proc. EuroGraphics 2023)

@article{cogalan23,
author = {Çoğalan, U. and Bemana, M. and Seidel, HP. and Myszkowski, K.},
title = {Video frame interpolation for high dynamic range sequences captured with dual-exposure sensors},
journal = {Computer Graphics Forum},
volume = {42},
number = {2},
pages = {119-131},
doi = {https://doi.org/10.1111/cgf.14748},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14748},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14748},
year = {2023}
}

Paper (Full Author's Copy)
									
				
Icon description Code

Materials

Video Tire Video Carpet Video Window Video Box Video Sphere Video Camera Shake

QVI

XVFI

FILM

Ours

Imprint / Data Protection