Image Restoration for Under-Display Camera

Yuqian Zhou         David Ren         Neil Emerton        Sehoon Lim        Timothy Large

IFP,UIUC         CIL,UC Berkeley         Applied Science Group, Microsoft

QMUL SurvFace

Abstract

The new trend of full-screen devices encourages us to position a camera behind a screen. Removing the bezel and centralizing the camera under the screen brings larger display-to-body ratio and enhances eye contact in video chat, but also causes image degradation. In this paper, we focus on a newly-defined Under-Display Camera (UDC), as a novel real-world single image restoration problem. First, we take a 4k Transparent OLED (T-OLED) and a phone Pentile OLED (P-OLED) and analyze their optical systems to understand the degradation. Second, we design a novel Monitor-Camera Imaging System (MCIS) for easier real pair data acquisition, and a model-based data synthesizing pipeline to generate UDC data only from display pattern and camera measurements. Finally, we resolve the complicated degradation using learning-based methods. Our model demonstrates a real-time high-quality restoration trained with either real or the synthetic data. The presented results and methods provide good practice to apply image restoration to real-world applications.

QMUL SurvFace

UDC Dataset

The new dataset is collected by a Monitor-Camera Imaging System (MCIS). We utilized 300 images form DIV2K dataset. For each display type: T-OLED, and P-OLED, we collected paired display-free and display-covered imaging data in the form of both 16-bit RAW sensor data, and 8-bit RGB. Images have resolution of 1024*2048. Because of the current challenge, we only release the paired RGB data for training. Please refer to our report for more details of the dataset.

News

Download

Under-Display Camera Dataset (RGB, 2.1G): (Due to the currently running challenge @ ECCV, we only provide training data now.) [Microsoft OneDrive] [Google Drive] [Baidu Drive (pw:9k7q)]

Citation

	    	Image Restoration for Under-Display Camera.
		Zhou, Yuqian and Ren, David and Emerton, Neil and Lim, Sehoon and Large, Timothy.
		Technical Report, 2020. Paper Bibtex
	    

Licence

UDC dataset can only be used for research purposes. All the images are collected from DIV2K dataset. The copyright belongs to Microsoft and the original owners.

Contact

Should you have any questions, please contact Yuqian Zhou via zhouyuqian133@gmail.com.