High-Dimensional Multivariate Linear Regression with Weighted Nuclear Norm Regularization

Abstract

We consider a low-rank matrix estimation problem when the data is assumed to be generated from the multivariate linear regression model. To induce the low-rank coefficient matrix, we employ the weighted nuclear norm (WNN) penalty defined as the weighted sum of the singular values of the matrix. The weights are set in a nondecreasing order, which yields the non-convexity of the WNN objective function in the parameter space. Although the objective function has been widely applied, studies on the estimation properties of its resulting estimator are limited. We propose an efficient algorithm under the framework of the alternative directional method of multipliers (ADMM) to estimate the coefficient matrix. The estimator from the suggested algorithm converges to a stationary point of an augmented Lagrangian function. Under the orthogonal design setting, the effects of the weights for estimating the singular values of the ground-truth coefficient matrix are derived. Under the Gaussian design setting, a minimax convergence rate on the estimation error is derived. We also propose a generalized cross-validation (GCV) criterion for selecting the tuning parameter and an iterative algorithm for updating the weights. Simulations and a real data analysis demonstrate the competitive performance of our new method. Supplementary materials for this article are available online.

Keywords:

• Generalized cross-validation
• Low-rank matrix
• Non-convex optimization
• Weighted nuclear norm

Supplemental Materials

More technical details and numerical results are summarized in the supplemental material. The code for the numerical results and figures of the article and the associated user guidelines are also available in the supplemental material.

Acknowledgments

We deeply appreciate the insightful suggestions and comments from the editor Dr. Faming Liang, the associated editor, and the two reviewers to improve our article. We also want to express our appreciation to Prof. Bin Li from the Department of Experimental Statistics at Louisiana State University for his kind discussions and suggestions for our article writing.

Disclosure Statement

The authors report there are no competing interests to declare.

Notes

1 Refer Negahban and Wainwright (2011) for checking how to translate MVLR to trace regression model. A description of the extended algorithm of WMVR-ADMM to trace regression is provided in Section G of supplemental material.

Additional information

Funding

This material is based upon work supported by the National Science Foundation under grant no. 2229876 is partly supported by funds provided by the National Science Foundation, the Department of Homeland Security, and IBM. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation or its federal agency and industry partners. The authors are also partially sponsored by NSF grants DMS 2015363 and the A. Russell Chandler III Professorship at Georgia Tech.