On the Use of Minimum Penalties in Statistical Learning

Abstract

Modern multivariate machine learning and statistical methodologies estimate parameters of interest while leveraging prior knowledge of the association between outcome variables. The methods that do allow for estimation of relationships do so typically through an error covariance matrix in multivariate regression which does not generalize to other types of models. In this article we proposed the MinPen framework to simultaneously estimate regression coefficients associated with the multivariate regression model and the relationships between outcome variables using common assumptions. The MinPen framework uses a novel penalty based on the minimum function to simultaneously detect and exploit relationships between responses. An iterative algorithm is proposed as a solution to the nonconvex optimization. Theoretical results such as high-dimensional convergence rates, model selection consistency, and a framework for post selection inference are provided. We extend the proposed MinPen framework to other exponential family loss functions, with a specific focus on multiple binomial responses. Tuning parameter selection is also addressed. Finally, simulations and two data examples are presented to show the finite sample properties of this framework. Supplemental material providing proofs, additional simulations, code, and datasets are available online.

KEYWORDS:

• Graph constrained models
• High-dimensional convergence
• Nonconvex optimization
• Post-Selection inference
• Selection consistency

Supplementary Materials

Proofs and Additional Simulations: An Appendix containing details for the theoretical results, additional simulations, and simulation results described in this article. (SuppMaterialSubmission.pdf)

Code and Datasets The R-Code and datasets along with documentation to reproduce simulations and examples described in this article. A readme file is supplied describing each file supplied. (submission_ code_ jcgs.zip)

Additional information

Funding

The project described was supported by the National Institute Of General Medical Sciences, 2U54GM104942-02 and National Institute On Minority Health And Health Disparities of the National Institutes of Health under Award Number U01MD017419. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. This work was also supported by the National Science Foundation Major Research Instrumentation Program Award 1726534.