How to solve checkerboard problem caused by input patches in deep learning based super resolution?
An answer to this question on Stack Overflow.
Question
I am studying a DBPN model, which is deep-learning model for the super resolution, with the tensorflow framework. I am suffered from "checkerboard problem" at scale x4 restoration. I know that the cause is using input patches (32x32 size) due to computing powers so that there is problem every 32 stride..
could anyone give me the solutions to cope with this problem ? Thanks!!
I already checked that kernel size should be multiple of the stride size
Answer
There's an excellent article about this on Distill. They state there:
One approach is to make sure you use a kernel size that is divided by your stride, avoiding the overlap issue. This is equivalent to “sub-pixel convolution,” a technique which has recently had success in image super-resolution [8]. However, while this approach helps, it is still easy for deconvolution to fall into creating artifacts.
Another approach is to separate out upsampling to a higher resolution from convolution to compute features. For example, you might resize the image (using nearest-neighbor interpolation or bilinear interpolation) and then do a convolutional layer. This seems like a natural approach, and roughly similar methods have worked well in image super-resolution (eg. [9]).
A quick Google search also reveals a number of papers and source code dealing with this issue.