AI Resume Screening: Same CV, 100 Runs, Scores 66 to 99

Is AI resume screening reliable enough to make hiring decisions on its own? HackerRank’s own numbers say no. The company open-sourced the AI agent it uses to screen applications, and feeding one unchanged resume through it 100 times produced scores from 66 to 99. At a typical 85-point cutoff, that identical candidate gets auto-rejected roughly two runs out of three.

AI screening tools are sold on two promises: speed and consistency. The speed is real. The consistency, it turns out, is not, and for once we don’t have to argue the point from theory, because the vendor published the code.

In late June 2026, Interview Street (HackerRank’s parent company) released hiring-agent on GitHub, the internal tool it built to triage its 50,000–60,000 annual intern applications. On July 1, Pinggy ran the test every buyer of AI screening should have run first: the same PDF, over and over, counting the spread.

This piece walks through what the test found, why the variance happens, and the questions we’d put to any AI screening vendor before letting their tool touch a hire/no-hire decision. We’re former software engineers who use AI in our own screening pipeline daily, so this is not an anti-AI argument. It’s an argument about where the judgement has to sit.

Key Takeaways

One unchanged resume, 100 runs through HackerRank’s open-sourced hiring-agent: scores ranged from 66 to 99 (Pinggy, July 2026).

At an 85-point cutoff, the identical candidate clears the bar roughly 35% of the time, rejected the other 65% on scoring noise alone.

65% of the score weights open-source and personal-project activity; production experience and technical skills together carry 35%.

Lowering the model temperature narrowed the spread (six runs: 27–34) but did not remove it.

The EU AI Act classifies recruitment and candidate-evaluation systems as high-risk, with obligations applying to these systems from August 2026.

What the AI resume screening test actually found

Pinggy’s method was simple: one resume file, unchanged, fed through the CLI repeatedly. The first four runs alone scored 90, 74, 88, and 83. By run 100, the full spread was 66 to 99, a 33-point band from identical input.

Here’s the arithmetic that matters for anyone using a score threshold:

Metric	Result
Runs	100, identical PDF
Score range	66–99 (33-point band)
First four runs	90, 74, 88, 83
Pass rate at an 85-point cutoff	~35%
Auto-reject rate, same candidate	~65%

Read that last row again. The candidate didn’t change. The resume didn’t change. Whether they advanced to a human depended on which run of the dice they happened to get.

The obvious engineering objection, “lower the temperature”, was tested too. Six consecutive runs at reduced temperature scored 27, 34, 32, 34, 34, and 30. Tighter, yes. Deterministic, no. The Hacker News thread on the release goes deep on why: temperature shapes the sampling distribution rather than eliminating it, and GPU floating-point behaviour adds non-determinism underneath that. Variance can be narrowed. It cannot be configured away.

This is the question we get from clients in some form every quarter: “can we just automate CV screening?” The honest answer has always been “partly.” Now there’s a public, reproducible dataset showing exactly which part.

Why the score moves: six LLM calls and a weighting problem

To be fair to HackerRank: the tool is MIT-licensed, openly documented, and was built to filter only the weakest applications from a 50,000-plus intern pipeline, with the majority advancing to human review. As an engineering artifact for a volume problem, it’s a reasonable design. The trouble starts when the same pattern gets applied to decisions it wasn’t built for.

The pipeline chains six separate LLM calls, one each for basics, work history, education, skills, projects, and awards, then enriches the result with GitHub data (classifying repositories, selecting the candidate’s top seven contributions) and scores everything against four weighted categories.

Two findings from that architecture deserve more attention than the headline variance number.

First, each LLM call is a fresh roll. Six chained probabilistic calls compound their individual noise. Pinggy found the technical-skills category behaved like a stable checklist, while the production-experience and project evaluations were volatile, subjective judgement calls. The parts of a career that most need interpretation are precisely the parts the model scores least consistently.

Second, and this is the structural problem, 65% of the total score weights open-source and personal-project activity. Production experience and technical skills together carry 35%.

For intern screening, that weighting is defensible: applicants have no production track record, so public code is the best available signal. Apply it past entry level and it inverts. A senior engineer with eight years on a closed-source enterprise platform, the profile behind most of the Cloud/DevOps and Data/AI searches we run, may have a near-empty public GitHub and a deep production record. This weighting scores that person on the 35% and largely ignores the rest. The tool isn’t measuring seniority. It’s measuring visibility.

What this means if you’re hiring past entry level

The lesson isn’t “AI screening is broken.” It’s that volume-screening logic doesn’t transfer to scarce-role hiring, and the failure is invisible unless you go looking for it.

The two problems compound at exactly the wrong end of the market:

Scoring variance costs you candidates you never see. At intern volume, a noisy filter that wrongly rejects some qualified applicants still leaves thousands. In a search where perhaps 30 people in the country genuinely fit the role, a ~65% false-reject rate at the threshold is not noise, it’s the search.
The weighting screens out the profile you’re actually hunting. Senior and staff-level engineers, particularly in enterprise and regulated environments, are systematically less visible on public GitHub. The scarcer and more senior the profile, the worse a projects-weighted score represents it. That’s why reaching passive candidates is a headhunting discipline, not a scoring exercise.

Our own split is the one we’ve written into how we work: AI handles the scale, sourcing reach and first-pass matching across thousands of profiles. The founders make the judgement calls on technical and role fit. Both matter. Neither replaces the other. HackerRank’s numbers are the clearest public evidence yet for why that boundary sits where it does: a 33-point spread is tolerable in a first-pass sourcing funnel with humans downstream, and indefensible as the deciding vote on a hire.

If you hire in the EU, this stopped being only a quality question this month.

Regulation (EU) 2024/1689, the EU AI Act, lists AI systems used for recruitment and selection, including filtering applications and evaluating candidates, as high-risk under Annex III. The obligations for those systems (risk management, logging, human oversight, accuracy requirements) began applying on August 2, 2026. A screening layer that scores the same input 66 one run and 99 the next is a documented accuracy and oversight problem under exactly that framework.

GDPR Article 22 was already there: candidates have the right not to be subject to decisions based solely on automated processing that significantly affect them. An 85-point auto-reject threshold with no human in the loop is the textbook case. The Hacker News discussion flagged both issues, alongside the ongoing US litigation against Workday over algorithmic screening, a signal of where enforcement attention is heading on both sides of the Atlantic.

We covered the employer obligations in detail in our guide to what the EU AI Act means for hiring. The short version: if an AI system scores or filters your candidates, you now own its accuracy, its audit trail, and its human oversight, whether you built it or bought it.

Five questions to ask before you trust an AI screening tool

Playbook time. Before any AI layer gets a vote on who your hiring managers see, put these to the vendor, or to your own team if the tool is internal:

“Run the same resume through it 20 times. What’s the spread?” If they haven’t measured run-to-run variance, they don’t know their own false-reject rate. HackerRank’s spread was 33 points; ask for theirs in writing.
“What are the category weights, exactly?” A 65/35 split toward public projects is a design choice with consequences. If the vendor won’t disclose weights, you can’t know who the tool systematically underscores.
“What happens at the threshold?” Hard auto-reject with no human review is both a quality risk and, in the EU, a GDPR Article 22 exposure. Insist on a band where borderline scores route to a person.
“Where’s the audit trail?” The EU AI Act’s high-risk obligations require logging and traceability. Per-decision evidence, not just a score, is the minimum.
“What profile was this calibrated on?” A tool built for 50,000 intern applications encodes intern-shaped assumptions. Ask what changes when it scores a staff engineer with 12 years of closed-source work.

A vendor with good answers to all five is worth talking to. A vendor with none is asking you to outsource judgement to a dice roll.

Frequently asked questions

Is AI resume screening accurate?

For parsing structured fields, mostly yes. For judging candidates, inconsistently. HackerRank’s open-sourced screening agent scored the same resume between 66 and 99 across 100 runs, and lowering the model temperature narrowed but didn’t remove the spread. Any accuracy claim should come with a published run-to-run variance figure.

Why did an AI give the same resume different scores?

LLM scoring is probabilistic sampling, not deterministic calculation. HackerRank’s tool chains six separate LLM calls per resume, and each call adds its own variance, with subjective judgements such as project quality and production experience fluctuating far more than checklist items like listed skills.

Should companies stop using AI to screen CVs?

No, they should stop letting it decide alone. AI is genuinely useful for sourcing scale and first-pass matching across large pools. The HackerRank data shows why the final call belongs to a human: at an 85-point cutoff, scoring noise alone rejected the identical candidate about 65% of the time.

Does the EU AI Act apply to AI resume screening?

Yes. Annex III of Regulation (EU) 2024/1689 classifies AI systems for recruitment and candidate evaluation as high-risk, with obligations covering human oversight, accuracy and logging applying from August 2, 2026. GDPR Article 22 separately restricts fully automated rejection decisions.

How is HackerRank’s hiring-agent different from a typical ATS?

A classic ATS filters on keywords and structured fields, crude but deterministic. hiring-agent replaces that with chained LLM evaluations plus GitHub analysis, which reads more context but introduces run-to-run variance a keyword filter never had. It also weights open-source and personal projects at 65% of the score, a choice built for intern screening.

The bottom line

AI resume screening just got its most honest benchmark, published by a vendor against its own tool: a 33-point spread on identical input, a ~65% false-reject rate at a standard cutoff, and a scoring rubric that weights public side-projects over production experience nearly two to one. None of that makes AI useless in hiring. All of it defines the boundary: scale is automatable, judgement is not, and as of August 2026, EU law expects you to know the difference.

What to take away:

Ask any screening vendor for their run-to-run spread. No number means no measured false-reject rate.
Check the weights before the demo impresses you. A projects-heavy rubric underscores senior closed-source engineers.
Never wire a hard auto-reject threshold. Route borderline scores to a person, in the EU especially.
Keep per-decision logs. High-risk obligations under the AI Act have applied since August 2, 2026.

If you’re hiring for the roles where this failure mode bites hardest, senior Cloud/DevOps and Data/AI engineers who don’t live on public GitHub, that judgement layer is exactly what we do. We’re former software engineers; the AI in our pipeline widens the funnel, and a founder makes the technical call on every shortlist. Brief your search and we’ll give you a straight read on what it takes to fill the role, including which parts of the process genuinely can be automated, and which can’t.

Data sources: Pinggy’s scoring-consistency test (July 1, 2026); interviewstreet/hiring-agent (GitHub, MIT license); Hacker News discussion; Regulation (EU) 2024/1689 (EUR-Lex). Last updated: August 15, 2026.