A supervised weeding method to cluster high dimensional predictors with application to job market analysis

Abstract

The clustering of high-dimensional predictors draws increasing attention in various scientific areas, such as text mining and biological data analysis. In standard clustering procedures, when predictors are clustered, they only showcase the inherent patterns within the predictor set, lacking the capacity to predict the response variable. To this end, a new supervised weeding algorithm is advocated to address the dual requirement of detecting sparse clusters and capturing the prediction effects. The proposed algorithm is based on an iterative feature screening and coherence evaluation procedure. It iteratively weeds off the unimportant predictors in a backward fashion, forming sequences of nested sets to determine data-driven optimal cut-offs. This study uses Monte Carlo simulation to assess the finite-sample performance of the proposed method. The findings demonstrate that both the clustering and prediction performance of the proposed method are comparable to existing methods that concentrate solely on one aspect of the dual targets. An analysis of a job description dataset is conducted to explore significant groups of keywords that affect employees' salaries.

Keywords:

• Supervised clustering
• high dimensional data
• feature screening
• clustering predictors
• text mining

Mathematical Subject Classifications:

• Data analysis
• big data
• high dimensional data analysis

Acknowledgments

The authors thank the editors and referees for their constructive comments and suggestions. The content is solely the responsibility of the authors. Furthermore, there are no competing interests to declare.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

Bi and Liu's research were supported by NNSFC 12271456, 71988101 and the Ministry of Education Research in the Humanities and Social Sciences 22YJA910002.