All cases Machine Learning

What makes a feed “good”? Evaluating recommendation algorithms with real people.

The Machine Learning team built four new feed algorithms. To choose between them, we first needed to know what “good” meant to the people using the app.

Company
Cyprus
Role
Senior UX Researcher
Period
2025 – 2026
Type
Evaluative

Context

Gameram is a social app where gamers find people to play with, available across the Americas. The feed is where those connections begin.

The Machine Learning team had built four new versions of the recommendation algorithm. Technical metrics help compare models, but they don’t tell you whether the result makes sense to the person on the other side of the screen.

Research question: what makes a feed “good” for Gameram users, and which of the four algorithms comes closest?

Approach

Before comparing the algorithms, we needed a yardstick. So the study started by defining criteria with people, and only then moved on to evaluation.

1. Interviews To understand, in users’ own words, what a good feed needs to deliver. The result was 12 criteria.
2. Moderated evaluation with users Each version of the feed was tested with users, who rated it and explained why.
3. Unmoderated evaluation with experts Internal experts evaluated the same versions, adding a second perspective on the results.
4. Combined analysis Ratings were combined with qualitative feedback to understand not just which version performed best, but why.

Outcome

4recommendation algorithms evaluated
12quality criteria defined with users
2perspectives: users and internal experts

The ML team gained user-defined criteria to compare feed versions, alongside technical metrics. Findings were shared in decision-focused reports with the CPO and the data science team.

Some details have been omitted for confidentiality.

Next case

Why was almost no one using the teammate finder?