Universal Inference Meets Random Projections: A Scalable Test for Log-Concavity

Abstract

Shape constraints yield flexible middle grounds between fully nonparametric and fully parametric approaches to modeling distributions of data. The specific assumption of log-concavity is motivated by applications across economics, survival modeling, and reliability theory. However, there do not currently exist valid tests for whether the underlying density of given data is log-concave. The recent universal inference methodology provides a valid test. The universal test relies on maximum likelihood estimation (MLE), and efficient methods already exist for finding the log-concave MLE. This yields the first test of log-concavity that is provably valid in finite samples in any dimension, for which we also establish asymptotic consistency results. Empirically, we find that a random projections approach that converts the d-dimensional testing problem into many one-dimensional problems can yield high power, leading to a simple procedure that is statistically and computationally efficient.

KEYWORDS:

• Density estimation
• Finite-sample validity
• Hypothesis testing
• Shape constraints

Supplementary Materials

Appendix: The appendix contains proofs of all theoretical results (Appendix A), additional simulations and visualizations for the two-component normal mixture setting (Appendix B), discussions on the relative power of full-dimensional and projection tests (Appendix C), simulations to test log-concavity when data arise from a Beta distribution (Appendix D), and additional details on the permutation test and trace test for log-concavity (Appendix E). (pdf file)

R code: The R code to reproduce the simulations and figures is available at https://github.com/RobinMDunn/LogConcaveUniv.

Acknowledgments

This work made extensive use of the R statistical software (R Core Team 2021), as well as the clustermq (Schubert 2019), data.table (Dowle and Srinivasan 2021), fitdistrplus (Delignette-Muller and Dutang 2015), kde1d (Nagler and Vatter 2020), ks (Duong 2021), LogConcDEAD (Cule, Gramacy, and Samworth 2009), logcondens (Dümbgen and Rufibach 2011), MASS (Venables and Ripley 2002), mclust (Scrucca et al. 2016), mvtnorm (Genz et al. 2021; Genz and Bretz 2009), and tidyverse (Wickham et al. 2019) packages.

Disclosure Statement

The authors report that there are no competing interests to declare.

Additional information

Funding

This work used the Extreme Science and Engineering Discovery Environment (XSEDE) (Towns et al. 2014), which is supported by National Science Foundation grant number ACI-1548562. Specifically, it used the Bridges system (Nystrom et al. 2015), which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC). RD is currently employed at Novartis Pharmaceuticals Corporation. This work was primarily conducted while RD was at Carnegie Mellon University. RD’s research was supported by the National Science Foundation Graduate Research Fellowship Program under Grant Nos. DGE 1252522 and DGE 1745016. AR’s research is supported by the National Science Foundation under Grant Nos. DMS (CAREER) 1945266 and DMS 2310718. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.