How HelloBetter Designed Their Interview Process Against AI Cheating
With the new process, they improved their time to hire, minimized the risk of AI cheating and focused more on practical programming fundamentals!
Korbit AI (Sponsored)
This week’s newsletter is sponsored by Korbit AI. Korbit is an AI-powered code review and engineering insights tool that helps software development teams deliver better code, faster.
It automates pull request (PR) reviews, identifies bugs and issues, and provides actionable feedback and recommendations → just like a senior engineer, but instantly and at scale.
Key Features:
AI-powered code reviews → PR Reviews within minutes and also understands the project context + learns from past reviews.
Automated PR summaries & descriptions → clear and concise PR descriptions, ensuring that reviewers have the right context + high-level summary of the review.
Engineering Management insights → Automatically generates release summaries, does security audits, tracks code quality + PR cycle time.
Korbit seamlessly integrates with GitHub, Bitbucket, and GitLab, making it a plug-and-play AI assistant that fits naturally into your workflow → no setup required.
Thanks to Korbit AI for sponsoring this newsletter. Let’s get back to this week’s thought!
Intro
There is a rise in AI-assisted tooling to help with job submissions and interviews.
Here is an example, posted by Dawid Moczadło not so long ago, where the interviewee actually changed their appearance with AI and responded to questions with answers generated by ChatGPT.
Dawid also mentioned that he could feel the GPT-4 bullet point-style responses.
The way he exposed the candidate was by asking them to take their hand and put it in front of their face, which would likely remove the AI filter. The candidate refused to do that and he ended the interview shortly after.
Such “attempts” are going to just be more common in my opinion and it’s important to design your interview process in a way that will minimize the success of such acts.
To help us with this, I am happy to have Anna J McDougall, Software Engineering Manager at HelloBetter, as a guest author for today’s newsletter article.
She is sharing with us the process she has led on re-structuring their interview process against AI cheating.
Let’s hand it over to Anna!
1. A Fairer, More Modern Approach to Technical Interviews
As someone obsessed with tech careers, business growth, and engineering management, I am constantly coming up against the boogeyman of technical interviews.
In many ways, tech companies are stuck in the thinking of 20–30 years ago: whiteboards, algorithms, logic puzzles, and tasks that revolved around an in-person interview space.
We’re in a new era, where 99% of engineering interviews are done virtually and the candidate’s workspace is uncontrollable by the interviewing team.
We need to adapt to this, not by simply porting previous methods across to an online platform, but by fundamentally rethinking the skills we expect 21st-century software engineers to have.
Naturally, opinions about what is wrong with technical interviews are prevalent both from employers and (potential) employees. For example, leetcode-style assessments have long been derided as being a bad test of abilities.
“Instead,” opponents argue, “you should use take-home tests or projects that the candidate can explain back to you”.
Ah, but you see, not everyone can afford to let their evenings and weekends disappear into these take-home tests or side projects.
Due to this inconveniently linear construct called “time”, more experienced engineers start having or caring for families, adopting pets, or (lord forbid) doing sports and hobbies. It’s also no big secret that the majority of housework and care duties still tend to be shouldered by women.
To put your best foot forward as a candidate, you will create the best possible solution, no matter how many hiring managers tell you to “only spend 2–3 hours on a solution”. Therefore, take-home or “own project” technical interviews tend to put older candidates and women candidates at a disadvantage.
One other problem popping up nowadays is the prevalence of AI tools for engineers to pass technical interviews, ranging from manually entering prompts into ChatGPT (now a bit old-school), all the way through to having someone else take the technical round for you and have your face artificially plastered onto theirs.
Going back to take-home tests for a moment, these are all but obsolete in the age of AI, in large part because all the solutions seem to be eerily similar to each other… 👀
So what do you do when…
Leetcode tests aren’t a good indication of the work itself.
Take-homes will favour those with more free time.
Asking freestyle technical questions lends itself to cheating with AI.
Here, I share our own journey at Berlin mental health DTx startup HelloBetter in transforming our technical interview round into a solution that not only mirrors the real work, but also doesn’t require an unreasonable amount of candidate time.
I don’t believe there is a perfect solution yet to the challenge of assessing technical skill in an engineering candidate, however, I do think this is the best solution I know of so far in terms of balancing realistic expectations, useful insights, and equitable testing conditions.
Background
In November 2024, we began overseeing changes to our hiring process by first defining what our current issues were.
Most of these will probably seem familiar to you:
An interview process with too many steps.
Long time-to-hire.
Not discovering crucial information about a candidate until late in the process (e.g. they want to move vertically or horizontally within a year)
Candidates using AI to cheat or misrepresent their abilities.
A technical interview round that feels more like a memory test.
Engineers’ time being taken up by interviewing candidates the hiring manager doesn’t want
Engineers who are confused when they recommend against a candidate in an interview round, but that candidate gets hired anyway, or vice versa.
Non-Technical Hiring Changes
Firstly, let’s look at the solutions we implemented that aren’t related to the technical interview:
Introduced a short questionnaire in the application form to screen out some common misalignments on goals, salary, etc. (Though it should be noted that all HelloBetter IC salary bands are publicly available)
Moved the technical round after the hiring manager round.
Combined our two final, half-hour interview rounds — the “Team Interview” and the “Founder Interview” — into one hour-long slot, to reduce delays from scheduling.
I offered more of my time and willingness, for example, to interview multiple candidates per day or back-to-back.
For our most recent hire, I made a point to stay in close contact with our recruiter, sharing at least 1–2 Slack messages every day to stay on top of the process, candidates, and next steps. We further had a half-hour sync meeting every week.
Most of the changes above were done in a collaboration between the Engineering Managers and Recruitment. I volunteered to take on the challenge of defining changes to the technical interview, in large part because I had a clear vision of how it could be done better.
We further had a half-hour sync meeting every week.
Most of the changes above were made in collaboration between the Engineering Managers and Recruitment. I volunteered to take on the challenge of defining changes to the technical interview, in large part because I had a clear vision of how it could be done better.
Technical Interview Round: Workshop Phase
I wanted to ensure this was something involving all of engineering. Since the engineers are the ones conducting the technical interview round, it made sense to have them drive the ideation and narrowing of the precise structure, based on my criteria.
In Slack, I asked for volunteers to assist me, and got 6 takers. By chance, they managed to neatly divide into two engineers each for mobile, web frontend, and backend.
The engineers in question were Julia Ero, Mohamed Gaber Elmoazin, Andrei Lapin, Caio Quinta, Euclides dos Reis Silva Junior, and Chaythanya Sivakumar.
I set up a Miro board and defined the goals/guidelines for what we wanted the technical round to achieve. In two sessions of about 1 hour each, I guided the engineers through questions and discussions about their own experiences both as candidates and as interviewers.
As you can see in the image, I provided a few guiding principles:
The goal of the interview is to provide the hiring manager with an idea of the technical abilities and level of the candidate, not to pass or fail that candidate.
Create an interview structure that can work across disciplines (i.e. backend and frontend interviews shouldn’t be wildly different)
The candidate should feel supported and welcome.
The process should be as close as possible to the actual work they will do, rather than on purely abstract concepts.
The end result should allow us to assess the candidate’s analytical capabilities, programming ability, understanding of programming principles, and “code communication” skills.
In the first session, we covered the engineers’ own experiences. The goal here was to ground their ideas in lived experience and to ensure that their framing began from the candidates’ perspective.
Each person was given their own post-it to fill out, and after each section, I would summarise what points seemed to be most common or agreed-upon.
From there, we devised the basic structure with a timeline of approximately 75–90 minutes total.
2. Introducing the McDougall Method
In this 90-minute interview structure, we assess the abilities of the candidate in a real-world scenario, which prioritises a fair process for the candidate with the pragmatic reality of a technical assessment.
The structure can be broken down into four phases:
Introduction to the candidate and a brief overview of their technical history (10–15 minutes)
Pair programming session (30–45 minutes)
Reflection/follow-up questions (15 minutes)
Candidate questions and wrap-up (10 minutes)
Separating each of these stages out, we had a combined brainstorming of what questions we could ask for each stage, and for Step 2 (Pair Programming), how we could best achieve the goals laid out above.
Each of these phases is important and serves a crucial purpose. Let’s look at each in turn.
A quick note on naming…
The term “McDougall Method” is my own label for the framework I led, developed and documented to address these technical interview challenges. I chose the name it’s more convenient than saying “our new technical interview method” or “HelloBetter’s updated technical interview round structure” on repeat.
However, like most things EMs take credit for, it was not actually a solo effort, and is thanks to the input of the six engineers mentioned above, as well as thanks to HelloBetter CTO Amit Gupta, and fellow EMs Leonardo Couto and Garance Vallat — who provided me with the freedom to drive this initiative.
Phase 1: Candidate Introduction and Technical Background
This phase is pretty common, but nevertheless, it’s important to understand why this phase exists in most technical interviews, rather than jumping straight into a coding exercise or a quiz. This stage only includes two engineers, to avoid the candidate being overwhelmed but also to avoid any potential individual bias playing too big of a role.
In short, we want the candidates to feel as relaxed as possible. We will never have fully relaxed candidates in this situation, but if there’s any way for us to reduce stress, we know that will lead to better performance. The brain does not operate at its best when it’s in fight-or-flight mode.
Here, we smile, introduce ourselves, make a bit of small talk about our roles at the company and maybe even a fun fact about ourselves.
We then give the candidate an opportunity to introduce themselves with the same basic questions they face in most interview stages, which should provide some ‘easy wins’ for them:
Tell us about yourself
Tell us about your recent work history
What excites you about the role?
…and so on.
We explicitly don’t want this stage to go on too long, again because of stress.
Every candidate enters the technical round knowing they will be assessed, so too much small talk is delaying the part they fear the most.
As such, we want enough introductory conversation to ‘grease the wheel’, but not so much that they start getting jittery about when the hard part will actually start. For us, we set this limit at about 10–15 minutes.
Phase 2: Pair Programming
This is probably the part of the post you came here to read, so let’s dive into it.
Firstly, I earlier stipulated that the coding portion of the interview should mimic the real job as closely as possible. For that, we required three things:
A codebase similar to our own: same tech stack, same naming conventions, same structure, etc.
Problems similar to ones we really fix.
A setup allowing the same flexibility and freedom of a real engineer.
The core of the McDougall Method is a 40-minute technical pair programming assessment providing a dummy codebase mirroring a section of your production code, complete with incomplete functions, faulty implementation, and failing tests.
The candidate is sent read-only access to the repository one hour before the interview, to allow them to look over it and get a feel for the code, how it’s structured, etc. However, they should not actively implement anything in that hour.
Once in the interview, the candidate pairs with one of the interviewing engineers, and together they work out how to solve (potentially) five problems of increasing difficulty.
The “Problem” of AI
“But Anna!” I hear you cry from behind your screen, “This doesn’t solve the problem of cheating with AI! They could have another screen, or another device, and be using AI to solve the problems!”
Well, we’ve managed to overcome this challenge by welcoming our new robot overlords, because the McDougall Method explicitly allows AI tools to be used.
That’s right. The candidate shares their screen, and is told (both in earlier communication and in the interview itself) that they can use Google, Stack Overflow, GPT, Copilot, Cursor, whatever they want. Most importantly, they can ask any questions they like of the interviewers, and should.
In the age of AI, what we want to assess in technical interviews is not how many facts or algorithms an engineer can memorize, but how they work, and if they understand what they create in the process.
If we accept that we want our engineers to use AI tools for productivity increases, then we have to view AI use as a skill. If our technical interview rounds are for assessing technical skill, then AI use must therefore be allowed.
Dos and Don’ts for the Codebase
I want to take a small pause here to cover some Dos and Don’ts for this phase.
Do:
Create a private repository that they can clone locally and easily install and run.
Use a small section of code, either taken from your production codebase or mirroring it closely.
Built in several levels of difficulty, for example, in frontend, you would start with a small styling change and build up to a complex, verbose function requiring full refactoring.
Don’t:
Use your actual production code.
Use candidates to complete real tickets (that is free labour and not cool).
Include any secrets or API Keys.
Forget to send the repository one hour beforehand.
On this last point, GitHub does not yet offer the feature of scheduling repository access. My hope is that this post and the ensuing avalanche of teams switching to the McDougall Method might inspire them to offer it as an option.
At present, the only way to do this is to schedule a reminder for yourself (or one of the interviewing engineers) one hour before the interview.
Technical Grading of the Exercises
I mentioned above that we use five different ‘levels’ of problem for each codebase, for example, a styling change as Level 1 and a full functional refactor at Level 5.
These levels should roughly correspond with a technical grading for the candidate. I include here some examples based on React as a framework:
Level 1: Intern or fresh junior (e.g. small styling change)
Level 2: Junior or low-level mid (e.g. onClick event not triggering the correct response)
Level 3: Mid-level ability (e.g. efficiently and accurately populating components from an API response)
Level 4: High-mid to low-senior ability (e.g. handling complex state management and/or re-rendering errors)
Level 5: Senior+ ability (e.g. full refactoring of complex verbose functions and/or inefficient component architecture)
Why? Because again, our goal here is not to see who our ‘best’ candidate is, but to see if the level we’re hiring into matches the candidate. This can also potentially avoid situations where a candidate underlevels themselves (again statistically more likely to occur with women), and ensures that we accurately determine what role to hire them into, which in turn assures fair pay, and lower mid- to long-term turnover.
When hiring for juniors, you would start at Level 1.
When hiring for mid-level engineers, you would start at Level 2.
When hiring for senior engineers, you would start at Level 3.
Why 40 minutes?
Put simply, because it’s enough time.
You aren’t going to get much more insight on a candidate that you haven’t gotten in 40 minutes of pair programming with them.
In truth, our final guidelines indicate a minimum of 30 and a maximum of 45, but 40 minutes seems to be the sweet spot.
Assessment Criteria
I’ve already indicated that coding ability and effective information sourcing (e.g. asking the colleague questions, using Google/AI, etc.) is the main goal of this round. However, also being assessed is their ability to communicate their ideas about how they’re solving the problem.
Here we ask the interviewing engineers to answer four questions:
How many tests did the candidate solve?
What was the average quality of the solutions they completed? (1–5 scale)
Could the candidate clearly communicate their decisions or approach?
What level would you estimate the candidate to be at?
We also give a free text area where they can go into more detail, for example about the tools used, any major errors or points of confusion, etc.
As much as I enjoy the structure of the first three questions, the free text box is usually where the most valuable information pops up, for example:
Candidates who spot errors in the code that were not on purpose (oops!)
Candidates who can’t explain how the results they get from AI/Google/etc work
Candidates who are unable to talk through their thought process
After that, we move into the next phase, which is directly related to the pair programming task.
Phase 3: Reflection / Follow-Up Questions
Almost as important as the pair programming session itself are the questions the team asks afterwards.
In a traditional technical interview round, the coding exercise itself is supposed to reveal the technical knowledge of the candidate. In the McDougall Method, the pair programming session is more about assessing how they work.
In Phase 3, the team has the job of asking questions, allowing the candidate to display how well they understand what they implemented.
As you can see, this phase itself has two steps.
Reflection
Firstly, we allow the candidate to reflect on how they feel. We know nerves are on edge, tensions are high, and candidates almost never perform at their best in an interview. So, let’s give them a chance to say so.
The first question every interviewing engineer should be asking before anything else in Phase 3 is:
How did you feel about that exercise?
This is an open question which allows them to explain their emotions, worries, fears; or even what they’re proud of. From there, questions around what they would have liked to do differently, or what other tasks they would have liked to do, can arise.
Follow-Up Questions
After this initial reflection, follow-up questions should be next. These revolve around why they chose a certain approach, what pitfalls might occur, how certain implementations would/could scale or be maintained; or perhaps questions about naming conventions, readability, or memory management.
In our latest round of interviews, it was here where good “implementers” had their knowledge gaps revealed. This is not necessarily a bad thing for the candidate because, as stated earlier, it is the hiring manager’s prerogative to match the candidate to the role.
For example, if you’re hiring a junior or a lower mid-level candidate, then you won’t expect many deep answers here, but knowing the limits of your hire will help with forming a development plan for their first year in the company.
In short, rather than assessing the core of their knowledge during the coding exercise in Phase 2, we allow the candidate to focus on pure implementation, and then go deeper in Phase 3.
Note: This is the section in which the engineers require the most EM guidance in terms of accurately defining the expected level of analysis/understanding.
Assessment Criteria
Due to how much this section can change, we opted to keep one question about candidate openness to feedback, but again, mainly relying on a free text summary for the interviewers to describe their approach, strengths, weaknesses, etc.
Phase 4: Candidate Questions and Wrap-Up
I don’t have much unique insight to add here, but I would say this is a crucial step to any interview. Some guidelines I provided to the engineers were:
Open the floor to the candidate for questions
When you think they’re done asking questions, ask one more time if there’s anything else they want to know
Explain the next steps (e.g. “We’ll pass our feedback to Anna and she will be in touch within the next few days”)
End with a smile and thank the candidate for their time
Assessment Criteria
As with Phase 1, we keep the assessment criteria for text information about what questions they asked and any final notes.
Providing Feedback About the Candidate
In addition to the fields and questions outlined above, our HR software Personio offers a “Negative”, “Neutral”, and “Positive” option for every candidate.
Our instructions for this final rating are as follows:
“Positive” should reflect an overall good experience with the candidate, not that you think they are technically suitable.
“Neutral” should be if the candidate was polite, but perhaps it was awkward or lacking cohesion, consistency, or decent communication ability.
Only leave a “Negative” if there is a serious cultural or interpersonal clash e.g. sexism, racism, aggression, cheating, etc. Do not select ‘Negative’ if you think they’re lower level than the position advertised.
In short, the final rating should act as a way to give us a ‘vibe check’. Personally, being aware of some of the issues around cultural evaluations, particularly for neurodivergent candidates, I would accept any candidate with Neutral or Positive if the other criteria are met.
Overall Evaluation Field
The last field is an open text field where each interviewing engineer can summarise their thoughts. In both our Confluence documentation and in the training sessions we emphasise the need to note both strengths and weaknesses in this field.
In the case of someone who really can’t complete the exercises or is totally underqualified, I would also expect the engineers to outline that here.
3. Outcomes of the McDougall Method
Improved time to hire
All of the changes taken above led to our “Days to Fill” metric being 30% of its usual in our latest hiring round. Naturally, this time equals money.
This metric was influenced by the streamlined interview process, the close work with our recruiter, and the engineers being trained to submit predictable, standardised evaluations.
In addition to the hours saved by moving the interview process along quickly, we also save time and mental energy from the context-switching of having to keep candidates in an overly-long pipeline: this goes both for the hiring manager and the interviewing engineers.
Lower chance of using AI to cheat and misrepresent their abilities
Although this is still technically possible, the separation of the coding phase from the analytical phase makes it harder to use AI. Questions are based around feelings, hypotheticals, and preferences, many of which have no right or wrong answer.
The ability to openly use AI during the pair programming exercise removes the element of ‘shame’ from using it as a tool.
Practical programming fundamentals are key
As a candidate, the McDougall Method requires zero memorisation or cramming of factoids. Instead, practical programming fundamentals become key.
If you’ve worked as a mid-level candidate at a company for the last three years, you will likely be levelled as a mid-level candidate with the McDougall Method, regardless of how recently you inverted a binary tree, how many JavaScript concatenation quirks you can recite, or how many HTTP status codes you can memorise.
The focus is on levelling candidates, not on pass/fail
The emphasis of this method is on levelling candidates and determining their strengths and weaknesses, NOT on giving them a pass/fail grade based on the interviewing engineers’ assumptions.
A candidate could be technically strong, but the cultural tradeoff might not be worth it when compared to a more technically-flawed candidate who is an excellent team player.
Furthermore, within the method itself, there is space for a difference between those who are implementers and those who are knowledge-havers. We all have a balance of both, and the structure allows us to see who is strong where, and what areas still need to be built up — crucial to building a development plan for a new starter.
In the end, it’s the hiring manager’s job to balance these considerations, and the McDougall Method gives ample opportunity for interviewing engineers to provide their feedback and build an accurate picture of a candidate’s technical abilities for that manager to weigh.
Future Action Points
There are still some ways to improve this method:
More clearly define the technical criteria for Phase 3: since this is highly dependent on the role, it is hard to devise concrete questions in a standardised format here.
More precise assessment criteria for Phase 3: as per the above, I think the variability in this phase means that we need to be open to more individualised assessment criteria here.
These two points lead me to believe that a new framework for the devising and assessment of this phase is required. For example, through some sort of question matrix or pre-hiring questionnaire to assess the relative importance of different technical insight areas and levels of knowledge per position.
Call to Action
Let’s keep things simple. My call to action is this…
Try it.
Take it to your EMs or your team and talk about it. What are the pros and cons? What might be missed?
If this resonates, share it with your team. I’m especially keen to hear what you’d do differently, whether here in the comments or on LinkedIn, where I regularly post about engineering management, tech careers, and my own observations about the industry and healthtech in particular.
Last words
Special thanks to Anna for sharing her insights on this important topic with us!
Make sure to follow her on LinkedIn and also check out her awesome talk on How to Turn Engineering Work into a Promotion, I highly recommend watching it if you are interested in growing in your career.
We are not over yet!
How to Thrive in Your First 90 Days in a New Role as an Engineer
Your first 90 days are crucial and can define your overall experience in your next position as an engineer. Learn how to nail your first 90 days as an engineer in my latest video!
New video every Sunday. Subscribe to not miss it here:
The Daily Byte
Everything you need to ace your coding interviews, from curated problems to real-time feedback and expert guidance. If you’re looking to level up for coding interviews, this would be a great place to start.
My friend Kevin has recently quit his job at Google to focus on this!
Liked this article? Make sure to 💙 click the like button.
Feedback or addition? Make sure to 💬 comment.
Know someone that would find this helpful? Make sure to 🔁 share this post.
Whenever you are ready, here is how I can help you further
Join the Cohort course Senior Engineer to Lead: Grow and thrive in the role here.
Interested in sponsoring this newsletter? Check the sponsorship options here.
Take a look at the cool swag in the Engineering Leadership Store here.
Want to work with me? You can see all the options here.
Get in touch
You can find me on LinkedIn, X, YouTube, Bluesky, Instagram or Threads.
If you wish to make a request on particular topic you would like to read, you can send me an email to info@gregorojstersek.com.
This newsletter is funded by paid subscriptions from readers like yourself.
If you aren’t already, consider becoming a paid subscriber to receive the full experience!
You are more than welcome to find whatever interests you here and try it out in your particular case. Let me know how it went! Topics are normally about all things engineering related, leadership, management, developing scalable products, building teams etc.
 | A guest post by
|
This is very good
With the tests in the pairing exercise, do you expect a senior to solve four of them and a senior+ all five of them? Or do you pick one or two based on level and just do those?