Promoting Diversity while tackling Popularity Bias in Two-Tower Recommender Systems

Abstract

Recommender systems, particularly Two-Tower architectures, often amplify popularity bias and create filter bubbles, limiting users’ exposure to diverse news content. This thesis investigates methods to promote con- tent diversity and mitigate popularity bias in news recommendation without significantly compromising personalisation accuracy. During this research I developed a baseline Two-Tower news recommender using a BERT-based NRMS architecture and evaluated it within a dynamic Simulation-Based Evaluation Framework designed to assess diversity and popularity bias over time, overcoming the limitations of static offline evaluation. The core contribution is a ”Diversity Awareness Second Chance” re- ranking mechanism. This method identifies older, long-tail articles from under-represented categories and uses a purpose-built User Choice Model (UCM) to validate their relevance before selectively swapping them into the final recommendation slate. Experiments show that our proposed method significantly outperforms standard diversification techniques like Maximal Marginal Relevance (MMR), which drastically reduces ranking accuracy (MRR drops from 0.778 to 0.241). The ”Diversity Awareness Second Chance” approach successfully increases the exposure of long-tail articles by 71.5% and achieves the highest diversity scores (Combined Diversity: 0.787), all while maintaining acceptable rank- ing performance (nDCG@10: 0.769) and avoiding the sharp accuracy drop associated with naive methods. This work demonstrates a practical and effective path toward building fairer, more diverse, and highly effective news recommender systems. The proposed mechanism, validated through a dynamic simulation, offers a con- figurable solution for balancing relevance with the crucial goals of fostering a well-informed and equitable information ecosystem.