Companies that give Take-Home Tests: review your hiring funnel

Take-home tests vs whiteboard coding are very opinionated in the tech community.

Some argue take-home challenges reproduce the real work environment with developer tools that anyone can do at their own convenience. The very same camp argues that whiteboard coding (algorithms + data structure) does not reflect daily routine. However the take-home approach has a huge time investment and the risk to be ghosted (ignored without feedback) is high.

The reasons why take-home assignments became popular is time invested asymmetry. It takes much less time to review than to do an assignment.
Same reason as why third-party recruiters love to phone call anytime. An email can be ghosted, a phone call cannot be ignored, the candidate has full attention to the recruiter.

Most of the take-home challenges are dummy applications that you could send from previously done take-home tests, yet those companies do not accept others and want you to complete theirs. Sometimes the take-home challenge is about a core of their business domain, it is code that they could use for free. Put another way, some companies consider the take-home test as crowdsourcing of free consulting.

Their hiring funnel looks like this

Here are the steps in the hiring funnel:

  • Applicant pool. The candidates who applied are in the applicant pool. The resumes and cover letter are usually read by a human in smaller companies who may call each candidate for the 1st interview.
  • HR: Major HR interview is a friendly 30 minutes chat. After everyone will receive a coding interview to complete at your home on your time. Some passive developers are contacted on various channels ( Linkedin, StackOverflow … ) by the HR/recruiter who try to bring attention.
  • Most recruiters have no way to filter technical people, usually almost everyone who interviewed with the recruiter will receive the coding challenge.
  • take-home challenge: Once the homework read, the majority of developers believe the challenge is not worth their time and will stop there. Among the candidates who interviewed first with HR and received the take-home test, how many of them complete the task? Few recruiters were transparent with me about the percentage of people who submit their code. My guesstimate is fewer than a third submitted if the company has an established productAs Gayle mentioned if this is screening round, it should be kept short.
  • interviews Often the code challenge is not even discussed, it was only used as a filter. On several occasions, I like that these interviews served as an open discussion for design choices and the take-home project serves as a template to add additional features during a live coding interview (googling is allowed)
  • hiring committee (if there is any). Interview reports are read and the salary range is discussed. An offer might match.

Pain points in this funnel

You can already see two major pain points in the funnel :

  • Almost everyone in the applicant pool was interviewed individually by HR. HR has to sell candidates to complete the homework. The HR is really busy to filter, interview, follow up with candidates. The focus is on quantity rather than quality.
  • bias on who completed the challenge. The hiring managers may believe that only successful candidates were the ones who submitted and may brush off that those who didn’t submit not were up to the challenges. In their vision, the take-home challenge is a test of worthiness for the next interviews.
  • Do you really think qualified candidates will do the test? People in demand will simply prioritise better alternatives.
  • very few candidates submit the take-home test. Because full-time employees who apply elsewhere prefer low hanging fruits. Didn’t you notice the HR keeps following the project status? If the conversion rate is low, something is wrong in the funnel.

Which leads to why don’t candidates just submit the homework?

Take-home challenge is not scalable

The take-home test approach is not scalable for both the interviewers and interviewees. As each project is unique, the interviewers have to reply to candidates who request clarity (if the authors ever reply at all). Reviewing the submitted project takes time to understand the logic.

For interviewees, I want to show the best quality of work I do. If I send a project that does all features with quick and dirty code, I am sure the reviewers will reject it outright. Reviewers expect a well-designed code structure. Ironically, the take-home project is only a temporary project which showcases your skills against the time limit. Yet, reviewers expect production quality.

Project scope & time underestimated

I don’t mind to complete an automated graded test online for 1-2h.

However, most take-home tests are the type “Build an app/website that X and Y from scratch. We expect tests, threading, SOLID architecture, deployment, production quality, offline mode … You have 4h (to unlimited time) to complete the project”.

Were the expectations and scope of the test stated properly? Often you have to navigate with ambiguity and the project scope is ridiculously too large.

I said I don’t mind to spend 2h. Oh now the time limit has been changed to 2h 😁 and the scope is still the same. Of course, the projects are highly underestimated.

You know what? Top companies offer to travel after successfully passing the HR and technical screening interviews (1h30 total). I (and anyone) know where to optimize time (our most limited resource).

As tech screening, online tests are much faster to evaluate candidates.

AUtomated tests as tech screening

Since the code challenge have to be reviewed by humans. It is much better to filter with online automated test (coderpad.io ) . Main advantages are:
partial grading. Expectations are known. Different reviewers have different way to grading.
time efficiency for the company. No one has to correct behind the scenes. Candidate

Why Top companies don’t use take-home tests?

Top tech companies FAANG (Facebook, Amazon, Apple, Netflix, Google) and other respectable ones in Europe ( Zalando, Criteo … ), Asia (Alibaba …) DON’T give take-home assignments as a prerequisite for entering into the process. There is no shortage of candidates willing to work for them who would take the assignment in a heartbeat.

After passing the HR interview (30 min) and first technical screening (45 min -1h), the candidate is offered to travel on-site (flight+hotel+food). So why spend more than 4h only on the possibility to reach interviews? While top candidates invested 1h30 interview time and are compensated with travel and accommodation.

Take-home assignment is a huge opportunity COST. Qualified candidates will be turned off. For screening, a 1h interview is enough. Serious companies know that the better candidates won’t put up with crazy requirements like a lengthy take-home that most likely lands in black hope.

I once explained this situation to an HR lady: “We are a start-up, we don’t have the resources”. In admitting this, she hinted the compensation will be lower.

There are way more people applying there. The ratio of candidates to interviewers is higher. How do these companies find the time to interview candidates?

Look at their hiring funnel. The hiring funnel picture below is not at scale. First, notice the applicant pool is much larger and filtered for the HR screening. Second, the take-home challenge is replaced by a short technical screening.

Optimal hiring funnel

  • much larger applicant pool. At large companies, it is not uncommon that candidate profiles are filtered by machines ( ATS = application tracking software) based on keywords. To stand out from a large number of people, you should ask an employee (who knows you) to refer you.
  • HR: usual same friendly HR interview who will guide about the whole process. HR will prepare you for the materials and resources to study to ace the next technical/behavioral interviews.
  • Technical screening (45min to 1h): introduction of candidate and interviewer followed by one whiteboard coding, sometimes two (if you do well). If you performed one working solution that is not the optimal ( time and space complexity), you will get a 2nd technical screening. Depending on the companies, the success pass ranges from 5 to 20%.
  • interviews. 3 to 5 interviews based on algorithms, system design, and behavioral. A bit more difficult than screening. Generally, you are invited on-site, flight and hotel are offered by the company. In the Coronavirus period, the interviews are online.
  • the hiring committee will read the interviewer report. Depending on the headcount and availability, they will make an offer.

How to improve the hiring funnel for everyone

By improving, I mean to save time for everyone. Notice the major differences between the two funnels. What companies that send take-home assignments ought to do:

  • Some companies claim that project review take more than 1h per engineer. Instead of assigning a take-home challenge after the HR interview, a technical screening of 1h (+10 minutes to write a report) is enough. In addition the candidate feels more appreciated to speak with human.
  • filter more aggressively the candidate pool. So recruiters will have more time to focus on quality rather than quantity. Some smaller companies have a drastic cut-off in screening. For example at 37signals, only the top 1-3% applicants are invited for take-home assignments after being selected based on their cover letter.
  • the screening interview is kept short. No BS take-home project for free.
  • if there is a take-home test, it MUST be short ( <2h), no ambiguity. Most take-home challenges, whose authors haven’t even done their own challenge, are vague.
  • If you really require a long take-home test, do at least one of the following:
    • Create a draft of this assignment and make sure one of your engineers ( who didn’t contribute to the assignment) can finish this project under the time limit. Make sure the authors themselves can finish their own assignments under half of the time limit. Make their project reviewed by another interviewer in a blind test. Oh, and it will great if you send this project model to anyone who submitted for a correction.
    • provide an existing code base to avoid boilerplate code. Most day-to-day job are not greenfield project. A large portion is fixing and building on top of existing code. The burden on reviewers is reduced as well.
    •  zero feedback is pathetic. Give a constructive feedback rather than Yes or No. Candidates read Glassdoor .
    • or narrow down the top candidates and pay them. Think about how much an external recruiter costs ( retainer fee + placement fee ). A few hundred USD/EUR multiplied by the reduced pool of candidates (max 10 ) is really cheap. I even saw an assignment “Here are the UI screens, make pixel-perfect adaptive UI workflow”. It looks like some free work you could use later.Make it clear, you need the work to be done and pay talent accordingly. I just laughed at this recruiter 😉.

Practice

From now on, I prefer to stick to whiteboard coding style interviews because they are time-boxed and scalable. Once I understand a problem, I never forget and re-apply the same principles elsewhere. The Leetcode efforts are amortized. I understand people who are willing to train for a few months on LeetCode/HackerRank and refuse to spend one week on a take-home assignment. On whiteboard coding , the metrics are clear: the solution passes all tests, most efficient in time and space complexity.

It is fun to experiment with a project with the latest libraries ( still in dev, alpha, beta version, I am looking at Android Hilt, Jetpack Compose … ). But it is better to understand the CS fundamentals used to build these libraries or frameworks. The algorithms and data structure are timeless knowledge.

Excuses for take-home assignments

With take-home assignments, you don’t know on what criteria you will be evaluated. Your application could be rejected on “unstated” trivia criteria:

– use the wrong formatting tool. In the developer circle, I heard various stories: “I spent 8h on an Android tic tac toe project, they turned me down because they didn’t like the code formatting”
– use a popular library that is not known by the team
– use an architecture that the team is not familiar with
– deliver a high quality project, over-engineered for the reviewer
– …

What are your experiences on the hiring funnel for any companies that send take-home projects? Please comment below.

If you are on the hiring side, please post your funnel numbers (how many received tests /submitted/were invited for interviews)

PS: from others experience, companies with take-home assignment are only “interviewing but not actually hiring”.

One thought on “Companies that give Take-Home Tests: review your hiring funnel

  1. Any company who compliments or declines a take-home assignment without sufficient feedback should be named and shamed !

    I don’t interview for companies that do lengthy take home challenge for this reason.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.