A Tale of Two Bases: Local-Nonlocal Regularization on Image Patches with Convolution Framelets

Inspired by recent high performance patch-based image inpainting algorithms using Point Integral Method (PIM) and Low Dimension Manifold Model (LDMM), we propose in this paper an image representation scheme combining the local and nonlocal characterization of patches in an image. By resolving an image as patches averaged out on their overlapping pixels, our representation scheme can be shown as equivalent to a tight frame constructed from convolving local bases (wavelet frames, discrete cosine transforms, etc.) with nonlocal bases (e.g. spectral basis induced by nonlinear dimension reduction on patches) and each component is called a {\it convolution framelet}. This insight leads to a novel interpretation of LDMM as a weighted $\ell_2$-regularization on the coefficients obtained by decomposing images into linear combinations of convolution framelets; based on this understanding, we extend the original LDMM to a reweighted version that yields improved inpainting results. In addition, we establish the energy concentration property of convolution framelet coefficients, where given a nonlocal basis the paired local basis is constructed from a linear reconstruction framework; a generalization of this framework to the setting of unions of local embedding is natural for interpreting BM3D, one of the state-of-the-art image denoising algorithms.