A few theoretical results for Laplace and arctan penalized ordinary least squares linear regression estimators

Abstract

Two new non convex penalty functions – Laplace and arctan – were recently introduced in the literature to obtain sparse models for high-dimensional statistical problems. In this article, we study the theoretical properties of Laplace and arctan penalized ordinary least squares linear regression models. We first illustrate the near-unbiasedness of the non zero regression weights obtained by the new penalty functions, in the orthonormal design case. In the general design case, we present theoretical results in two asymptotic settings: (a) the number of features, p fixed, but the sample size, $n\to \infty$, and (b) both n and p tend to infinity. The theoretical results shed light onto the differences between the solutions based on the new penalty functions and those based on existing convex and non convex Bridge penalty functions. Our theory also shows that both Laplace and arctan penalties satisfy the oracle property. Finally, we also present results from a brief simulations study illustrating the performance of Laplace and arctan penalties based on the gradient descent optimization algorithm.

KEYWORDS:

• arctan penalty
• non convex penalties
• Laplace penalty
• regularization
• variable selection

Acknowledgements

We thank the two anonymous reviewers for suggestions that led to improvement of the paper.

Additional information

Funding

Majnu John’s research was supported partially by the following National Institute of Mental Health grants: R34MH120790 (PI: Birnbaum), R21MH122886 (PI: Barber), R01MH117646 (PI: Lencz), R61MH123574 (PIs: Cornblatt/Carrion), R01MH120313 (PI: Deligiannidis), R01MH120594 (PIs: Kane/Robinson), R01MH108654 (PI: Malhotra), R01MH117172 (PI: DeChoudhury).