Expert Rating System | PO Task

A document enabling the development team to begin MVP implementation without additional business logic clarifications | By Yevhenii Holovei

PO Test Task

MVP

Rating Algorithm

01.

Problem

User Problem

Subscribers cannot objectively evaluate tipster quality. Without a unified rating, they follow popular rather than effective experts -- and lose money.

Business Problem

Without a transparent quality metric the portal cannot differentiate tipsters. Reputational risk grows. Advertising monetization of profiles is limited without proven effectiveness.

02.

Scope

In scope (MVP)

Out of scope

03.

Rating Algorithm Approach

Approach Comparison

Approach	Pros	Cons	Verdict
Win Rate Leaderboard	Simple to implement \| Easy to explain \| Intuitive	Small samples (2/2 = 100%) \| Ignores odds \| Incentivizes favorites	Rejected
Bayesian ROI Score	Reflects real value \| Neutralizes small samples \| Manipulation-resistant	Harder to explain \| Sensitive to data quality \| Requires tuning	Selected

Rationale

Betting audiences care about profitability, not just win share. Win Rate without odds is easy to manipulate. Bayesian correction solves the small-sample problem without a hard cut-off threshold.

04.

Algorithm Details

Win / Lose / Push

The system uses a flat betting unit model. Each prediction is treated as exactly 1 unit staked.

Outcome	profit_i
WIN	`odds_at_publish − 1`
LOSE	`−1`
PUSH	`0` (counts toward N, lowering ROI)

ROI_rawROI_raw = Σ(profit_i) / N

Example: 2 win (+2.50, +0.80), 2 lose (−1.00, −1.00), 1 push (0.00) → Sum = +1.30, N = 5 → ROI_raw = 0.26

How Odds Are Factored In

Bet	Odds	Outcome	Profit
A	1.2	WIN	+0.2
B	3.5	WIN	+2.5
C	5.0	LOSE	−1
D	1.1	LOSE	−1

Bets C and D each lose exactly −1 despite vastly different odds. An expert with 90% WR at avg_odds = 1.2 will have a low ROI -- the system naturally penalizes manipulation via "safe" bet selection.

Small Sample (Bayesian Correction)

SCORE Formula

PRIOR_N = 10
Confidence = N / (N + PRIOR_N)
SCORE = Confidence × ROI_raw

N	Confidence	Effect
2	2/12 = 0.17	Even 100% WR yields Score ≈ 0.39
50	50/60 = 0.83	Rating almost entirely reflects actual ROI

Public leaderboard display threshold: N ≥ MIN_BETS = 5. Profiles with N < 5 show "insufficient data" status and are excluded from the rankings table.

Anti-Manipulation Measures

published_at ≥ match_datetime

Late Bets

Record rejected → rejection_log (reason='late_bet'). >5 rejections within 30 days → moderator flag.

Multiple predictions for the same match

Duplicates

Only the first is kept (MIN published_at). Others → rejection_log (reason='duplicate'). >3 in 7 days → flag.

Systematic low-odds betting

Favorites

ROI naturally penalizes low-odds strategies. No separate flag implemented in MVP.

05.

Database

expert_ratings Table Fields

Field	Type	Example	Note
`expert_id`	VARCHAR(50)	`"C"`	PK
`n_valid_bets`	INTEGER	`9`	For Confidence
`wins`	INTEGER	`7`	For Win Rate on UI
`losses`	INTEGER	`2`	Audit
`pushes`	INTEGER	`0`	Reserved
`sum_profit`	DECIMAL(10,4)	`6.0400`	For ROI
`roi_raw`	DECIMAL(10,4)	`0.6711`	For UI and audit
`score`	DECIMAL(10,4)	`0.3179`	Sort basis
`avg_odds`	DECIMAL(5,2)	`2.30`	For profile
`rank`	INTEGER	`2`	For quick lookup
`is_qualified`	BOOLEAN	`TRUE`	Public TOP filter
`last_calculated_at`	TIMESTAMP	`2026-03-20 06:00 CET`	Freshness

Computed on-the-fly (not stored)

win_rate (wins/n_valid_bets) | confidence (n_valid_bets/(n_valid_bets+10)) | relative ROI growth over the last 30 days

06.

Decomposition

Epic: Expert Rating System

As a portal marketing manager, I want a transparent expert ranking based on proven profitability, so I can increase user trust and incentivize activity from qualified tipsters.

User Stories

Rating Score Calculation

As the system, I should daily compute a SCORE for each qualified expert so the rating always reflects current standing.

Public Leaderboard Page

As a portal visitor, I want to see a top-ranking of qualified tipsters sorted by SCORE, so I can choose the best one to follow.

Expert Profile with Metrics

As a user, I want to see detailed stats (SCORE, WR, ROI, N, rank) on a tipster profile, so I can assess effectiveness before subscribing.

Manipulation Detector ★ Ready-for-Dev

As an internal moderator, I want to automatically receive flags when suspicious activity is detected (late bets, duplicates), so I can review manipulation in a timely manner.

Acceptance Criteria -- Manipulation Detector

Given / When / Then

Late Bet Detection

GIVEN: Cron job triggered the data preparation pipelineWHEN: published_at ≥ match_datetime of the relevant matchTHEN: record is rejected → rejection_log (expert_id, prediction_id, reason='late_bet', detected_at=NOW())AND: late_bet flag count ≥ 5 within 30 days → moderation_flags severity='high'

Duplicate Detection

GIVEN: Pipeline is processing prediction expert_id=X, match_id=MWHEN: a valid prediction with the same expert_id and match_id already existsTHEN: new record is rejected → rejection_log reason='duplicate', reference_pred_idAND: duplicate count ≥ 3 within 7 days → moderation_flags severity='medium'

Edge Cases

published_at = match_datetime

EC 1

Exactly at match start → reject (late, non-strict ≥ inequality)

Match cancelled after publication

EC 2

event_status = 'cancelled' → rejection_log reason='event_cancelled'. Do not count as manipulation.

Two predictions with identical published_at

EC 3

Keep the one with the lower prediction_id. Log both in rejection_log.

match_datetime is NULL or invalid

EC 4

Reject both predictions with reason='invalid_event_data'. Do not generate a manipulation flag.

Bot-like: 100 predictions per second

EC 5

rapid_submission_count ≥ 10 within 60 seconds → moderation_flags severity='critical'

Definition of Done

rejection_log receives a correct entry for every rejected prediction from all 4 preparation steps

moderation_flags are generated under threshold conditions (≥5 late/30d, ≥3 dupes/7d)

Unit tests cover all 5 edge cases

API endpoint GET /moderation/flags returns a paginated list of flags

07.

Integration & Edge States

Where the Rating is Displayed

/leaderboard

Public page, only is_qualified = TRUE, sorted by SCORE

/expert/{id}

Profile: rank, SCORE, WR, ROI, N, avg_odds, last_calculated_at

Home Page

Top-3 block (widget)

Email / Push

Notification on rank change of ±3+ positions (out of MVP scope, reserved)

Edge States

Situation	System Behavior
New expert N=0	Profile without rating metrics. Badge "Insufficient data (0 predictions)". Not included in leaderboard.
N < MIN_BETS (1–4)	SCORE is calculated and stored in DB, but is_qualified = FALSE. Profile shows "Accumulating (N of 5)".
Missed data update	etl_status check → if pipeline incomplete, skip recalculation. Retain previous rating + error alert.
All events upcoming	Recalculation does not change the rating. last_calculated_at is preserved from the previous day.
Expert deleted/blocked	is_qualified = FALSE; soft delete (active=FALSE). Disappears from leaderboard, row is retained.

08.

Non-Functional Requirements

Performance & Scale

Metric	Target
GET /leaderboard	< 200 ms with 1000 qualified experts (Redis, TTL 1 hr)
GET /expert/{id}	< 100 ms (DB read with index on expert_id)
Batch calc for 1000 experts	< 5 minutes
Predictions per year	1000 tipsters × 365 days × 5 predictions = 1.8M rows
expert_ratings	1 row per expert, < 1 MB
rejection_log	Up to 20 rows per prediction ≈ 30K rows/month

Risks

MediumUpdate Delayetl_status check

Fallback to previous rating

LowManipulationrejection_log + flags

Late bet and duplicate detector

HighTuning PRIOR_NDB parameter

Change without deploy

PO Test Task -- Expert Rating System | Author: Yevhenii Holovei | Scope: MVP | Bayesian ROI Score Algorithm