Interview Other Engineers — Career Growth for Engineers, Part 8
Career Growth for Engineers · Part 8
A hiring debrief for a senior backend role. Four interviewers, one candidate. The first to speak is the most senior engineer in the room: "Strong hire. Great energy, we really clicked, reminded me of how I approached problems early on." Two of the other three had written "lean no hire" in their heads, both for the same reason: the candidate never tested their design against failure, and waved away a question about data consistency. Neither says so firmly. The candidate is hired. Six months later their first major design ships without retries or idempotency, and the team spends a quarter cleaning it up.
Nothing in that loop was malicious. The interview was unstructured, the strongest impression was social, the feedback was given out loud in order of seniority, and the two accurate signals were drowned out. Each of those is a fixable process problem.
The working claim: interviewing is a measurement task, and it gets better with the same things that improve any measurement: a defined target, a consistent instrument, and independent readings. Your personal sense of a candidate is one of the least reliable signals in the room.
Before you start
Get the role's level guide and the job description, and ask the hiring manager which two or three capabilities this loop most needs to establish.
Find out which part of the loop is yours (coding, design, behavioural, domain) and what the other interviewers are covering, so you don't all test the same thing.
Find or create the scorecard for your interview. If none exists, Step 3 shows how to write one.
Step 1: Know why structure matters (10 min)
The evidence on interview formats is unusually consistent. In 2022, Sackett, Zhang, Berry and Lievens re-examined decades of meta-analyses on how well selection methods predict job performance, correcting a statistical overcorrection in earlier estimates. In their revised figures, structured interviews came out as the strongest predictor they examined, ahead of general cognitive ability tests. Levashina and colleagues' 2014 review defines what "structured" means in practice: the same job-related questions for every candidate, defined rating scales, and scoring done by the interviewer before discussion.
Unstructured interviews persist anyway, because they feel informative. Dana, Dawes and Peterson (2013) showed that interviewers formed confident impressions even when a candidate's answers were random, and preferred having an unstructured interview to not having one, even when it made their predictions worse. "We really clicked" is the feeling that research describes.
Test you can run: after your next interview, before looking at your notes, write your hire/no-hire call and the one reason for it. Then check whether that reason is about evidence for a capability the role needs, or about how the conversation felt. If it is the second, structure will improve your calls.
Step 2: Prepare the same core questions for everyone (15 min)
Write your questions before the interview, tied to the capabilities you were asked to assess, and ask the same core questions of every candidate for the role. Follow-ups can vary; the core can't, or you can't compare candidates.
For engineering interviews, two kinds of question work well:
Work-sample questions: a scaled-down version of real work, such as designing a service the team actually built, or reviewing a pull request with planted problems.
Past-behaviour questions: "Tell me about a time a design of yours failed in production. What happened, and what did you change afterwards?" Ask for specifics: dates, numbers, what they did rather than the team.
Test you can run: read your questions back and, for each one, name the capability in the level guide it measures. Any question you can't map is testing something the role doesn't need, or something you just enjoy asking.
Step 3: Score against anchors, not against yourself (15 min)
A scorecard turns impressions into comparable evidence. For each capability, write what a weak, adequate and strong answer looks like at this level, in observable terms. For a senior design interview:
Capability | 1 – Below bar | 3 – At bar | 5 – Strong |
Failure handling | Doesn't consider failure unless prompted | Identifies main failure modes and retries when asked | Designs for failure unprompted: idempotency, timeouts, partial failure |
Trade-offs | Picks one option with no alternatives | Compares two options on one dimension | Compares options on cost, risk and operability, and explains the choice |
Clarifying the problem | Starts building immediately | Asks some scoping questions | Pins down requirements, scale and constraints before designing |
The anchors stop you from scoring against your own preferences, such as "solved it the way I would". Score each row during or immediately after the interview, with a one-line piece of evidence for each score.
Test you can run: give two interviewers the same recorded or written answer and have them score it independently with your scorecard. If their scores differ by more than one point on any row, the anchor for that row is too vague. Rewrite it.
Step 4: Watch for the predictable distortions (15 min)
A few biases show up in almost every engineering loop:
Similarity: rating candidates higher when they think or talk like you. "Reminds me of me" is a warning sign, not a compliment.
Halo: one strong area (a famous former employer, an elegant first answer) colours every other score.
Stress mistaken for ability: being watched while solving a problem raises anxiety, and that affects performance in ways unrelated to the job. Part 9 covers the research. As an interviewer, reduce the artificial pressure: explain the format, allow thinking time, and judge the reasoning, not the speed.
Order effects: the candidate after a very weak one looks stronger than they are.
Test you can run: for each score below 3 you give this month, check whether you wrote down specific evidence or a general impression. Impressions without evidence are where these biases live.
Step 5: Write feedback before the debrief, and make it usable (10 min)
Submit your written scores and evidence before you hear anyone else's view. Kahneman, Sibony and Sunstein's Noise (2021) describes how judgements made after discussion converge on whoever spoke first or loudest, and lose the independent information each person had. Independent readings first, discussion second, is the single cheapest improvement to most hiring loops.
Usable feedback has three parts per capability: the score, the evidence ("asked about retries only after I prompted twice"), and what it means for the role. Avoid "not a culture fit" without specifics; it is unfalsifiable and is often similarity bias in disguise.
Test you can run: reread your last piece of written feedback as if you were a hiring committee member who never met the candidate. Could you make the decision from it? If it is mostly adjectives, rewrite it with evidence.
Step 6: Disagree in the debrief, with evidence (10 min)
If your read differs from the room's, say so, with the evidence from your scorecard. A loop where everyone agrees immediately is often a loop where people deferred. A practical rule many teams use: the most junior interviewer speaks first, so seniority can't anchor the discussion.
Test you can run: in your next debrief, notice who speaks first and whether anyone changes their stated view after hearing them. If views shift without new evidence being presented, propose the junior-first rule.
Worked example: the same loop, run two ways (composite scenario)
Context. A regional bank's platform team hires two senior engineers in consecutive quarters. The first loop runs as described in the opening: unstructured, verbal feedback in seniority order.
Decision. After the cleanup quarter, the team lead redesigns the loop. Each interviewer gets one capability, a scorecard with anchors like the table above, and two fixed core questions. Written scores are due before the debrief, and the most junior interviewer speaks first.
Reasoning. The lead's diagnosis was that the earlier loop had the right signals, two interviewers had spotted the gap, but the process discarded them.
What happened. In the second loop, one candidate charms the senior interviewer again, who writes "strong hire" but scores failure handling at 2, with the evidence "didn't raise retries until asked twice". Seeing their own written evidence, the senior interviewer changes their recommendation before the debrief. The team hires a different candidate whose scores are consistent across all four interviewers. The new hire's first design review includes failure handling unprompted.
Lesson. The interviewers didn't become better judges. The process stopped throwing away what they had noticed.
Trade-offs to make deliberately
Choice | You gain | You risk |
Fixed core questions | Comparable evidence across candidates | Feeling scripted; keep follow-ups free |
Anchored scorecards | Consistent scoring, less bias | Time to write and maintain the anchors |
Written feedback before debrief | Independent readings, less anchoring | Slightly slower decisions |
Work-sample questions | Closer to the real job | Preparation time, and the need to refresh questions that leak |
Common failure modes
Hiring for rapport. "We clicked" predicts liking, not performance.
Different questions for every candidate. You end up comparing conversations, not evidence.
Verbal feedback in seniority order. The most senior view becomes the decision.
"Not a culture fit" with no evidence. Usually similarity bias.
Testing speed under pressure. It measures anxiety as much as ability.
Final takeaway
Interviewing well is less about reading people and more about measuring consistently: the same core questions, anchored scores with evidence, written down before anyone else speaks. Your next action: before your next interview, write the anchors for the two capabilities you are assessing, and submit your scores before the debrief starts.
Sources
Sackett, P. R., Zhang, C., Berry, C. M. and Lievens, F. (2022). Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range. Journal of Applied Psychology, 107(11), 2040–2068.
Levashina, J., Hartwell, C. J., Morgeson, F. P. and Campion, M. A. (2014). The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature. Personnel Psychology, 67(1), 241–293.
Dana, J., Dawes, R. and Peterson, N. (2013). Belief in the Unstructured Interview: The Persistence of an Illusion. Judgment and Decision Making, 8(5), 512–520.
Kahneman, D., Sibony, O. and Sunstein, C. R. (2021). Noise: A Flaw in Human Judgment. New York: Little, Brown Spark.
Continue the Career Growth for Engineers path
Previous — Part 7: Build a Network
Part of the Career Growth for Engineers learning path. Related: Technical Interviews From Both Sides and Hiring Engineers in an AI World.
