Gibbs Sampling for Mixtures in Order of Appearance: The Ordered Allocation Sampler

Abstract

Gibbs sampling methods are standard tools to perform posterior inference for mixture models. These have been broadly classified into two categories: marginal and conditional methods. While conditional samplers are more widely applicable than marginal ones, they may suffer from slow mixing in infinite mixtures, where some form of truncation, either deterministic or random, is required. In mixtures with random number of components, the exploration of parameter spaces of different dimensions can also be challenging. We tackle these issues by expressing the mixture components in the random order of appearance in an exchangeable sequence directed by the mixing distribution. We derive a sampler that is straightforward to implement for mixing distributions with tractable size-biased ordered weights, and that can be readily adapted to mixture models for which marginal samplers are not available. In infinite mixtures, no form of truncation is necessary. As for finite mixtures with random dimension, a simple updating of the number of components is obtained by a blocking argument, thus, easing challenges found in transdimensional moves via Metropolis-Hastings steps. Additionally, sampling occurs in the space of ordered partitions with blocks labeled in the least element order, which endows the sampler with good mixing properties. The performance of the proposed algorithm is evaluated in a simulation study. Supplementary materials for this article are available online.

Keywords:

• Dirichlet process
• Pitman-Yor process
• Size-biased permutations
• Species sampling models
• Stick-breaking construction

Acknowledgments

The authors are grateful to an Associate Editor and two Referees for their helpful comments and suggestions. The article was completed while M. F. Gil–Leyva was a Postdoctoral Fellow at Bocconi University (2021–2022) and at University of Torino (2022). This author is also thankful for the partial support of PAPIIT-UNAM project IG100221.

Disclosure Statement

The authors report there are no competing interests to declare.

Supplementary Material

The supplementary material available online includes the proof of Theorem 1; an example on how to compute the normalization constant of (15); details on the Geometric and Exchangeable Stick-Breaking process priors; examples on how to compute the set of admissible moves of d_i, and the acceleration step of Section 3.3; and an extension of the simulation study for the DP model. Additionally, we provide R and Julia codes and the datasets.

Additional information

Funding

P. De Blasi is supported by the Italian Ministry of Education, University and Research (MIUR) through PRIN 2015SNS29B and through “Dipartamenti de Eccellenza” grant 2018–2022.