2D to Immersive video conversion
Immersive video is experiencing a growing interest for some time now. Despite large investments from industraial giants, one problem remains that prevents immersive video from being adopted on a wider scale; the lack of real content. The vast majority of current immersive content is synthetic, generated by computer graphics models. The only approach for generating real content is through the use of complicated hardware set-ups. Such set-ups are composed of multiple cameras stacked next to each other. Using the hardware set-ups require upgrading the entire production pipeline. This, however, is an expensive solution to be adopted on any large scale.
In this project we present an automated system for converting the commonly available 2D video content into a multi-view, sterescocpic 3D format. Current conversion methods porduce low-quality video results that exhibit artifacts. Such artifacts are not acceptable to many viewers. Our system is largely data-driven; we leverage the power of big data through the latest advances in A.I. and deep neural networks. This allows us to generate high quality conversion for a wide spectrum of video content.
Motivated by Qatar’s hosting of the 2022 soccer World Cup, we pay a special attention to processing soccer videos. Our system is also capable of handling a wider spectrum of sports games and generic content such as movies, TV programs and others.
- K. Calagari, M. Elgharib, P. Didyk, A. Kaspar, W. Matusik, and M. Hefeeda, “Gradient-based 2D-to-3D Conversion for Soccer Videos”, ACM Multimedia (MM’15), Brisbane, Australia, October 2015.
- K. Calagari, M. Elgharib, S. Shirmohammadi, and M. Hefeeda, “Sports VR Content Generation from Regular Camera Feeds”, ACM Multimedia (MM’17), San Francisco, California, October 2017.
- K. Calagari, M. Elgharib, P. Didyk, A. Kaspar, W. Matusik, and M. Hefeeda, “ Data Driven 2D-to-3D Video Conversion for Soccer “, IEEE Transactions on Multimedia, 2017.
- A. Nandoriya, M. Elgharib, C. Kim, M. Hefeeda, and Wojciech Matusik, “ Video Reflection Removal through Spatio-temporal Optimizaation”, IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 2017.
- A. Hassanien, M. Elgharib, A. Selim, S. Bae, M. Hefeeda, and W. Matusik, “ Large-Scale, Fast And Accurate Shot Boundary Detection Through Spatio-Temporal Convolutional Neural Networks “, CoRR abs/1705.03281 (2017).
- S. Bae, M. Elgharib, M. Hefeeda, and W. Matusik, “ Efficient And Scalable View Generation From A Single Image Using Fully Convolutional Networks”, CoRR abs/1705.03737 (2017).
- Gradient-based 2D to 3D Video Conversion, US Provisional Patent Application, Filed August 2015.