ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 1308.6273
136
206
v1v2v3v4v5 (latest)

New Algorithms for Learning Incoherent and Overcomplete Dictionaries

28 August 2013
Sanjeev Arora
Rong Ge
Ankur Moitra
ArXiv (abs)PDFHTML
Abstract

In sparse recovery we are given a matrix AAA (the dictionary) and a vector of the form AXA XAX where XXX is sparse, and the goal is to recover XXX. This is a central notion in signal processing, statistics and machine learning. But in applications such as sparse coding, edge detection, compression and super resolution, the dictionary AAA is unknown and has to be learned from random examples of the form Y=AXY = AXY=AX where XXX is drawn from an appropriate distribution --- this is the dictionary learning problem. In most settings, AAA is overcomplete: it has more columns than rows. This paper presents a polynomial-time algorithm for learning overcomplete dictionaries; the only previously known algorithm with provable guarantees is the recent work of Spielman, Wang and Wright who gave an algorithm for the full-rank case, which is rarely the case in applications. Our algorithm applies to incoherent dictionaries which have been a central object of study since they were introduced in seminal work of Donoho and Huo. In particular, a dictionary is μ\muμ-incoherent if each pair of columns has inner product at most μ/n\mu / \sqrt{n}μ/n​. The algorithm makes natural stochastic assumptions about the unknown sparse vector XXX, which can contain k≤cmin⁡(n/μlog⁡n,m1/2−η)k \leq c \min(\sqrt{n}/\mu \log n, m^{1/2 -\eta})k≤cmin(n​/μlogn,m1/2−η) non-zero entries (for any η>0\eta > 0η>0). This is close to the best kkk allowable by the best sparse recovery algorithms even if one knows the dictionary AAA exactly. Moreover, both the running time and sample complexity depend on log⁡1/ϵ\log 1/\epsilonlog1/ϵ, where ϵ\epsilonϵ is the target accuracy, and so our algorithms converge very quickly to the true dictionary. Our algorithm can also tolerate substantial amounts of noise provided it is incoherent with respect to the dictionary (e.g., Gaussian). In the noisy setting, our running time and sample complexity depend polynomially on 1/ϵ1/\epsilon1/ϵ, and this is necessary.

View on arXiv
Comments on this paper