HackerRank open-sourced its resume-scoring agent; analysis finds the scores non-deterministic

Category: Engineering post
Status: discussion
Sources: analysis, repository, discussion
Summary: HackerRank published an open-source LLM resume-scoring agent (interviewstreet/hiring-agent, MIT, about 3,250 stars). An analysis ran one resume 100 times through the default gemma3:4b model at temperature 0.1 and recorded scores from 66 to 99 out of 120; at an 85-point cutoff the same candidate would be rejected about 65 percent of the time. The author traces the variance to the subjective project- and experience-scoring prompts, which carry no rubric, examples, or anchors, while the checklist-based technical-skills score stays stable across runs.
Why it matters: It is a concrete measurement of how LLM-as-judge scoring conflates reliable parsing with unreliable evaluative judgment, a failure mode for any team wiring a language model into automated screening or grading.