Auto-augmentation with Differentiable Renderer for High-frequency Shape Recovery

Research output: Chapter in Book/Report/Conference proceedingConference contribution


We propose a technique to estimate a high-resolution depth image from a sparse depth image captured by depth camera and a high-resolution shading image obtained by a RGB camera using deep neural network (DNN). In our technique, the network model is pretrained by synthetic images which are generated by rendering high-frequency shapes created by arithmetic model, such as sinusoidal wave of wide variation of parameters. Although the preparation of an appropriate synthetic dataset is critical for such tasks, it is not trivial to find a compact and optimal distribution of shape parameters. In this paper, we propose an auto augmentation technique to optimize hyperparameters for shapes achieving minimum number for training DNN. The proposed augmentation network directly optimizes the hyperparameters of a 3D scene including parameters of procedural shapes and their positions by gradient descent algorithm via a differentiable rendering technique. Unlike previous data augmentation techniques which only have basic image processing methods, such as affine and color transformations, the proposed method can generate optimal training dataset by changing the 3D shape and its position by using a differentiable renderer. In our experiments, we confirmed that our method improved the accuracy of high-resolution depth estimation as well as efficiency of training the network.

Original languageEnglish
Title of host publication2022 26th International Conference on Pattern Recognition, ICPR 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)9781665490627
Publication statusPublished - 2022
Event26th International Conference on Pattern Recognition, ICPR 2022 - Montreal, Canada
Duration: Aug 21 2022Aug 25 2022

Publication series

NameProceedings - International Conference on Pattern Recognition
ISSN (Print)1051-4651


Conference26th International Conference on Pattern Recognition, ICPR 2022

All Science Journal Classification (ASJC) codes

  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Auto-augmentation with Differentiable Renderer for High-frequency Shape Recovery'. Together they form a unique fingerprint.

Cite this