High-dimensional partial correlation coefficients: A survey study of estimation Methods

Abstract.

The partial correlation coefficient (Pcor) measures the degree of association between two random variables after removing the effect of another set of variables, called the controlling variables. When the dimension of these controlling variables is high, estimating Pcor becomes exceedingly challenging, as conventional computational methods fail to provide efficient estimators. Although several methods have been proposed for estimating Pcor in high-dimensional data, they primarily focus on testing whether its value is zero rather than enhancing estimation efficiency. This article concentrates on improving estimation efficiency and surveys existing methods. Through extensive simulation studies on high-dimensional data, we observe that all current methods consistently underestimate the absolute values of Pcor, with the bias being more pronounced when Pcor is positive. By considering a combination of regression equations, $X=Z\alpha +\epsilon$ and $Y=Z\beta +\zeta$, the method proposed in this article demonstrates a significant improvement over existing approaches. Stocks data in Hong Kong is employed to demonstrate the effectiveness of these methods.

Keywords:

• High-dimensional data
• estimation methods
• regularization methods
• partial correlation

Acknowledgments

The authors would like to thank the editor and two reviewers for their constructive comments, which helped to improve the article. The authors would also like to thank Professor Yingcun Xia for his invaluable guidance throughout the research.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research is supported by the National Natural Science Foundation of China (72033002).