Vetting Flow Iteration | Lemon.io
Problem inventory, root-cause analysis of Mark's 50% false-positive rate, and a 4-week hardening plan | By Yevhenii Holovei
Problem
Lemon.io runs a voice AI screener called Mark (provided by vapi.ai) to qualify developers before human review. The target is 150 hires/month. The current result is 104 -- not because of headcount, but because Mark generates more rework than it eliminates.
- Precision failure. Mark's auto-advance passes 50% false positives -- every other candidate it clears fails human QC. Recruiters must manually review every single Mark Pass. Automation frees zero capacity.
- Rate leak. Mark shares internal compensation ranges in 74% of calls. Candidates anchor to the ceiling instead of stating their own number -- eroding an estimated $4–5K/month in offer margin.
- Funnel bleed. Only 54% of candidates complete the Mark call. 208 drop due to tech failures with zero retry logic. Another 38% drop at stage 2 alone ("project preferences") due to call UX.
Net effect: recruiter utilization sits at 53% -- not from lack of people, but from AI-generated rework. The 150/month target is structurally blocked until Mark's precision is fixed.
Solution
Problem Inventory & Prioritization
Framework: Impact × Confidence × Urgency / Effort
Where: Impact = effect on 4 business metrics (throughput: hires/month, recruiter capacity, candidate trust, margin per hire) · Confidence = is the problem proven? · Urgency = will the impact grow if we delay? · Effort = engineering weeks (2 BE + 1 FE + 1 QA, no designer)
| # | Problem | What suffers | Evidence | Effort | Priority |
|---|---|---|---|---|---|
| 1 | Mark Pass is broken: 50% of auto-advance candidates downgraded after recruiter QC. (English / role-claims / skill-probe stages). | Recruiter capacity, throughput vs 150/mo goal | 15 of 30 reviewed downgraded to Reject (50%): Olena: 5 cases of Mark Pass with A2 English in a single week; Yulia: Mark passes scripted answers (3+ similar caught in standup notes) | ±4 weeks | P0 Top |
| 2 | Mark discloses the internal rate range to candidates (74%). | Margin per hire, ~5–6 lost offers/mo | Maryna: 17 of 23 cases (74%) in the past 30 days. After hearing the range, candidates push for the top of it instead of their original ask. 5–6 lost offers based on Maryna's estimate. | ±0.5 day (prompt change) | P0 Quick-win |
| 3 | Mark call completion only 54% (hangups 27% + tech issues 13% + agent-ended 6%). Biggest drop (38%) on stage 2/8 -- "project preferences". | Top-of-funnel conversion, candidate trust | 736 lost out of 1600 · candidate quotes call it "pushy" · no retry after tech failures (208 candidates) | ±2–3 weeks (stage-2 prompt fix + retry for tech-issue) | P1 |
| 4 | Need Review queue overflow: 160 of 400 waiting up to 14 days. No SLA, FIFO, no prioritization. | Candidate trust, time-to-matching | Olena: "queue grows by 20–25/day, we clear 8–10" · Kostyantyн: "I don't know the priority order" | ±1–2 weeks (scoring engine + SLA) | P1 |
| 5 | Mark Reject false positives: 73% override rate on Mark Reject + Coderbyte Pass. Mark is overly strict on pauses and re-asks. | Throughput (false rejects) | 11 of 15 overridden · Andrii: 8-sec pause → Mark flagged "shady" | ±1 week (Reject criteria + regression test) | P1 Low volume |
| 6 | Flagged for Decline (FfD) queue is just rubber-stamping: 16 h/mo on manual confirmation, while override rate is only 7%. | Recruiter capacity | Andrii: 2.5 h/day; 4 screens per candidate | ±2 days (bulk-confirm UI + filter) | P2 Quick-win |
| 7 | QC UI: no filter by routing decision, no stage indexing in transcripts. Recruiter spends more time finding the call than doing the QC itself. | Recruiter capacity | Yulia + Maryna feedback; Maryna gave up on QC mid-way | ±1 week (routing filter + stage indexing) | P2 Quick-win |
| 8 | Voice/TTS + content quality: robotic, "Cuber-NEE-ties", "React Jessup", "Redux in Node.js". No retry mechanism on the 208 tech-issue calls. | Candidate trust, top-of-funnel | 12 "stressful" vs 16 "loved it" (mixed); 8 "call dropped, can't redo" | ±1 day (pronunciation + question QA) | P2 |
Why #1 = Mark Pass precision (and not Rate leak)
Root Cause of Top Priority
Mark Pass precision 50% -- Hypothesis
Mark isn't "slightly miscalibrated". 3 of its 8 stages are fundamentally broken: English check, role/seniority claims, and skills probe. This isn't a threshold problem -- it's a missing rubric and no cross-validation against the CV.
| # | Stage | Status |
|---|---|---|
| 1 | Intro | English check is a hidden sub-check within all stages but first signals are here |
| 2 | Project preferences | UX issue (38% drop) |
| 3 | Role clarification | Broken -- No Claims vs CV check |
| 4 | Skills verification | Broken -- Mark accepts templated answers |
| 5 | Rate discussion | Leak (shares internal range -- Problem #2) |
| 6 | Availability & logistics | Ok |
| 7 | Soft skills probe | Ok |
| 8 | Close | Ok |
Chain of Reasoning: Mark's 3 Broken Stages
English check isn't a named stage in the spec -- the check is implicit, with first signals captured at Intro. By default, Voice LLMs judge "answered vs. didn't answer," not fluency level. Without a 3-tier rubric (A1-2 / B1 / B2+) tied to conversational triggers, Mark doesn't catch the level of English. → Olena: 5 Mark Pass cases with A2 English in one week.
Mark receives structured profile data from the parsed CV, but the prompt doesn't require comparing "candidate claims senior" against graduation_year + employment history. → Olena: a candidate claiming senior 2 years after graduation with zero prior experience passed.
Yulia caught a Mark Pass on: "Yes, I have extensive experience in React. I have worked on many React projects." Keyword + affirmative tone → Pass, with no follow-up probe ("give me an example of a bug you fixed," "a trade-off you faced"). Cooperative agent rewards a lot of text > quality.
Quantitative Confirmation
Auto-advance (AI-trusted fast-track) = Mark Pass + Coderbyte Pass + clean scam score. Of the 60 auto-advance cases, 30 were QC'd → 15 downgraded to Reject (50% precision). That's ±30 false positives per month × 20 min = ±10h of wasted recruiter time, plus extra load on the downstream tech interview.
50% advance rate -- no QC has been run on these. Yulia: 9 out of 10 (??) Mark Unclear cases were just candidates asking to repeat the question. True precision is likely similar to auto-advance.
Mark Reject + Coderbyte Pass is overridden 73% of the time, while Mark Pass + Coderbyte Reject is overridden only 8%. Mark is simultaneously too strict on edge cases and too lenient on the main path. This isn't a shifted threshold -- it's a noisy decision boundary. Fixed with a rubric, not by moving the cutoff.
Counter-Evidence (Considered and Rejected)
The 30 of 60 samples may have been hand-picked as "suspicious" rather than drawn at random. Validation: in week 1, run random QC across the full auto-advance population to validate the baseline.
No. It tests coding ability, which is orthogonal to the English / seniority / skill-probe failure modes.
Candidate quotes objectively confirm templated answers and fake seniority -- these aren't recruiter judgment calls, they're observable failure modes.
Why this blocks the umbrella goal
What I'd Build or Change
Focus: Mark Evaluation and Rubric Hardening
Phase 1: Evaluation -- week 1 (1 BE + QA)
Phase 2: Targeted Fixes -- weeks 2–3 (parallel tracks)
Phase 3: Shadow + Ship -- week 4
Metrics We Track
| Metric | Baseline | 4-week target | Quarterly target | Why |
|---|---|---|---|---|
| Auto-advance precision (post-QC pass rate) | 50% | ≥80% | ≥90% | North-star |
| Mark Reject precision (1 − override rate on Mark Rej + Coderbyte Pass) | 27% | ≥75% | ≥90% | Anti-bias |
| Mark Unclear precision (% Unclear = true) | ~10% (??) | ≥60% | ≥80% | Queue load → |
| Recruiter hours per vetted dev | 1.3h | 1.1h | ≤1.0h | Capacity |
| Hires/month | 104 | 120 | 150 | Umbrella |
Dependencies and Risks
Without it, there's no way to validate before shipping. If engineering tells us it's "3 weeks, not 3 days" → we ship with shadow-only validation (worse, but workable).
If the current model doesn't support tool-calling, Track B (CV cross-reference) gets harder. Mitigation: structural guardrail outside the LLM (at the routing layer, on the parsed CV) -- a proven approach.
All changes are prompt + backend + non-visual rubric. Recruiter-side UX improvements live in the quick-wins (no designer needed).
Bonus: Quick-Wins to Ship Alongside
QW-1. Rate-Leak Prompt Fix (1 BE × 0.5d + QA 0.5d)
What: Update system prompt with explicit policy: "Never disclose rate ranges or typical numbers. If asked, deflect: 'Rate is part of the offer discussion with the recruiter. What rate are YOU looking for?'" Add a regression test on 23 known-case transcripts (Maryna's pull).
Metric: % of calls with rate disclosure. Target <2% within one week (sample 30 calls manually, then keyword detector across transcripts).
Impact
QW-2. Flagged for Decline Bulk-Confirm + Sticky Filter (1 FE × 2d)
What: Checkboxes next to each case + "Confirm selected" button + keyboard shortcuts ('C' to confirm, 'O' to override to review). Filter "show only DISQUAL, scamScore 90–100". Override rate is only 7% → 93% of flags are correct, safe to accelerate.
Metric: Median time per Flagged for Decline case. Target <1 min (down from 4 min). Saves ±12h/mo of recruiter time = capacity for ±30 additional Need Review cases.
QW-3. Need Review Prioritization + Soft SLA (1 FE + 1 BE × 3d)
What: Add a priority formula that sorts cases by two signals -- how long they've waited + how likely they are to advance (Mark Pass + Coderbyte Pass = high). Recruiter sees high-priority cases at the top of the queue, with a "Likely Advance" badge on easy wins. Candidates waiting more than 5 days get an automatic email: "Still under review, expect an update by [date+3]" -- a trust signal, not a hard SLA.
Metric: Median time-in-queue <5 days; <10% of cases waiting >7 days. Lift in candidate satisfaction.
One Surprise + One Question Before Day 0
Surprise
Bonus: stage 2, "project preferences," has the highest drop (38%) -- even though it's the most neutral stage in the call. Mark pushes for specifics, and flexible candidates walk away. A conversion leak in an unexpected place.
Question to the team on Day 0
Without an eval set, we can't measure precision/recall on any change -- every "prompt improvement" becomes wishful thinking. Bonus question: what's Mark's LLM / voice stack, and does it support tool-calling? That determines the feasibility of Track B.
What I Don't Know
Random or cherry-picked as suspicious -- affects the 50% baseline. Validation: run random QC across the full auto-advance population in week 1.
Yulia's "9 of 10" is too small; we need categorization across 100+ cases.
Affects how we prioritize retry-on-tech-issue.
Needs segmentation by channel, otherwise baseline drop-off is partly explained by it, not by Mark.