117
206

New Algorithms for Learning Incoherent and Overcomplete Dictionaries

Abstract

A matrix ARn×mA \in \R^{n \times m} is said to be μ\mu-incoherent if each pair of columns has inner product at most μ/n\mu / \sqrt{n}. Starting with the pioneering work of Donoho and Huo such matrices (often called {\em dictionaries}) have played a central role in signal processing, statistics and machine learning. They allow {\em sparse recovery}: there are efficient algorithms for representing a given vector as a sparse linear combination of the columns of AA (if such a combination exists). However, in many applications ranging from {\em sparse coding} in machine learning to image denoising, the dictionary is unknown and has to be learned from random examples of the form Y=AXY = AX where XX is drawn from an appropriate distribution --- this is the {\em dictionary learning} problem. Existing proposed solutions such as the Method of Optimal Directions (MOD) or K-SVD do not provide any guarantees on their performance nor do they necessarily learn a dictionary for which one can solve sparse recovery problems. The only exception is the recent work of Spielman, Wang and Wright which gives a polynomial time algorithm for dictionary learning when AA has {\em full column rank} (in particular mm is at most nn). However, in most settings of interest, dictionaries need to be {\em overcomplete} (i.e., mm is larger than nn). Here we give the first polynomial time algorithm for dictionary learning when AA is overcomplete. It succeeds under natural conditions on how XX is generated, provided that XX has at most k \leq c \min(\sqrt{n}/\mu \log n, m^{1/2 - \epsilon}) non-zero entries (for any ϵ>0\epsilon > 0). In other words it can handle almost as many non-zeros as the best sparse recovery algorithms could tolerate {\em even if one knew the dictionary AA exactly}.

View on arXiv
Comments on this paper