Sequential Data Integration Under Dataset Shift

Abstract

With the rapidly increasing availability of large-scale and high-velocity streaming data, efficient algorithms that can process data in batches without requiring expensive storage and computation resources have drawn considerable attention. An emerging challenge in developing efficient batch processing techniques is dataset shift, where the joint distribution of the collected data varies across batches. If not recognized and addressed properly, dataset shift often leads to erroneous statistical inferences when integrating data from different batches. In this article, two shift-adjusted estimation procedures are developed for updated estimation of the parameter in the presence of dataset shift. Under prior probability shift, we can obtain parameter estimation and assess the degree of dataset shift simultaneously. We study the asymptotic properties of the proposed estimators and evaluate their performance in numerical studies. The proposed methodologies are illustrated with an analysis of the Ford GoBike docked bike-sharing data.

KEYWORDS:

• Batch processing
• Data integration
• Dataset shift
• Large-scale data
• Pairwise pseudolikelihood

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was partially supported by the Young Scientists Fund of the National Natural Science Foundation of China No. 12201616 (Sheng), and National Institutes of Health grant R01CA193888 (Huang).