Widely used deep latent variable models (DLVMs), in particular VariationalAutoencoders (VAEs), employ overly simplistic priors on the latent space. Toachieve strong clustering performance, existing methods that replace thestandard normal prior with a Gaussian mixture model (GMM) require defining thenumber of clusters to be close to the number of expected ground truth classesa-priori and are susceptible to poor initializations. We leverage VampPriorconcepts (Tomczak and Welling, 2018) to fit a Bayesian GMM prior, resulting inthe VampPrior Mixture Model (VMM), a novel prior for DLVMs. In a VAE, the VMMattains highly competitive clustering performance on benchmark datasets.Integrating the VMM into scVI (Lopez et al., 2018), a popular scRNA-seqintegration method, significantly improves its performance and automaticallyarranges cells into clusters with similar biological characteristics.