MLmediumProduction failure~12m

Newsfeed Model Offline Online Gap

Problem

Your newsfeed ranking model shows 0.85 AUC on offline holdout data, but online A/B tests reveal engagement is down 12% compared to the previous model. The offline metrics look great - precision@10 improved from 0.73 to 0.81, and NDCG@50 is up 6%. However, users are spending less time on the platform and daily active users dropped 3% in the test group. What's your diagnostic approach?

Reference solution

Reference solution available after you attempt the question.

Ready to solve it?

Start a session on Mockbit #94. You'll get graded with specific critique when you submit.