Block size, parallelism and predictive performance: finding the sweet spot in distributed learning

Abstract

As distributed and multi-organization Machine Learning emerges, new challenges must be solved, such as diverse and low-quality data or real-time delivery. In this paper, we use a distributed learning environment to analyze the relationship between block size, parallelism, and predictor quality. Specifically, the goal is to find the optimum block size and the best heuristic to create distributed Ensembles. We evaluated three different heuristics and five block sizes on four publicly available datasets. Results show that using fewer but better base models matches or outperforms a standard Random Forest, and that 32 MB is the best block size.

KEYWORDS:

• Distributed machine learning
• distributed file system
• hadoop
• machine learning

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by FCT – Fundação para a Ciência e Tecnologia within projects [grant number UIDB/04728/2020], [grant number EXPL/CCI-COM/0706/2021] and [grant number CPCA-IAC/AV/475278/2022.