Synthetic Population Testing for Recommendation Systems

Offline evaluation is necessary for recommender systems. It is also not a full test of recommender quality. The missing layer is not only better aggregate metrics, but better ways to test how a model behaves for different kinds of users before launch. TL;DR In the last post, I argued that offline evaluation is useful but incomplete for recommendation systems. After that, I built a small public artifact to make the gap concrete. In the canonical MovieLens comparison, the popularity baseline wins Recall@10 and NDCG@10, but the candidate model does much better for Explorer and Niche-interest users and creates a very different behavioral profile. I do not think this means “offline evaluation is wrong.” I think it means a better pre-launch evaluation stack should include some form of synthetic population testing: explicit behavioral lenses, trajectory-aware diagnostics, and tests that make hidden tradeoffs visible before launch. What Comes After “Offline Evaluation Is Not Enough”? In the fi