Correspondence Analysis (CA)-the method of choice to analyze contingency tables-is widely applied in text analysis, psychometrics, chemometrics, etc. But CA becomes difficult to interpret when items load on several dimensions, when dimensions comprise items whose loadings are of intermediate values, or when the number of rows or columns is large-a configuration routinely found in contemporary statistical practice. For principal component analysis (PCA), this interpretation problem has been traditionally handled with rotation and more recently with sparsification methods often inspired by the LASSO. Curiously, despite the strong connections between CA and PCA, sparsifying correspondence analysis remains essentially unexplored. In this paper, we extend the Penalized Matrix Decomposition (a relatively recent method based on the singular value decomposition) to sparsify CA. We present some theoretical results and properties of the resulting sparse correspondence analysis and illustrate this method with the analysis of a large textual data set.