Companies that give Take-Home Tests: review your hiring funnel

Take-home tests vs whiteboard coding are very opinionated in the tech community.

Some argue take-home challenges reproduce the real work environment with developer tools that anyone can do at their own convenience. The very same camp argues that whiteboard coding (algorithms + data structure) does not reflect daily routine.

Most of the take-home challenges are dummy applications that you could send from previously done take-home tests, yet those companies do not accept others and want you to complete theirs. Sometimes the take-home challenge is about a core of their business domain, it is code that they could use for free. Put another way, some companies consider the take-home test as crowdsourcing of free consulting.

Their hiring funnel looks like this

Here are the steps in the hiring funnel:

  • Applicant pool. The candidates who applied are in the applicant pool. The resumes and cover letter are usually read by a human in smaller companies who may call each candidate for the 1st interview.
  • HR: Major HR interview is a friendly 30 minutes chat. After everyone will receive a coding interview to complete at your home on your time. Some passive developers are contacted on various channels ( Linkedin, StackOverflow … ) by the HR/recruiter who try to bring attention.
  • take-home challenge: Once the homework read, the majority of developers believe the challenge is not worth their time and will stop there. Among the candidates who interviewed first with HR and received the take-home test, how many of them complete the task? Few recruiters were transparent with me about the percentage of people who submit their code. My guesstimate is fewer than a third submitted if the company has an established productAs Gayle mentioned if this is screening round, it should be kept short.
  • interviews Often the code challenge is not even discussed, it was only used as a filter. On several occasions, I like that these interviews served as an open discussion for design choices and the take-home project serves as a template to add additional features during a live coding interview (googling is allowed)
  • hiring committee (if there is any). Interview reports are read and the salary range is discussed. An offer might match.

Pain points in this funnel

You can already see two major pain points in the funnel :

  • Almost everyone in the applicant pool was interviewed individually by HR. HR has to sell candidates to complete the homework. The HR is really busy to filter, interview, follow up with candidates. The focus is on quantity rather than quality.
  • bias on who completed the challenge. The hiring managers may believe that only successful candidates were the ones who submitted and may brush off that those who didn’t submit not were up to the challenges. In their vision, the take-home challenge is a test of worthiness for the next interviews.
  • Do you really think qualified candidates will do the test? People in demand will simply prioritise better alternatives.
  • very few candidates submit the take-home test. Because full-time employees who apply elsewhere prefer low hanging fruits. Didn’t you notice the HR keeps following the project status? If the conversion rate is low, something is wrong in the funnel.

Which leads to why don’t candidates just submit the homework?

Take-home challenge is not scalable

The take-home test approach is not scalable for both the interviewers and interviewees. As each project is unique, the interviewers have to reply to candidates who request clarity (if the authors ever reply at all). Reviewing the submitted project takes time to understand the logic.

For interviewees, I want to show the best quality of work I do. If I send a project that does all features with quick and dirty code, I am sure the reviewers will reject it outright. Reviewers expect a well-designed code structure. Ironically, the take-home project is only a temporary project which showcases your skills against the time limit. Yet, reviewers expect production quality.

Were the expectations and scope of the test stated properly? Often you have to navigate with ambiguity and the project scope is ridiculously too large.

Project scope & time underestimated

I don’t mind to complete an automated graded test online for 1-2h.

However, most take-home tests are the type “Build an app/website that X and Y from scratch. We expect tests, threading, SOLID architecture, deployment, production quality, offline mode … You have 4h (to unlimited time) to complete the project”.

I said I don’t mind to spend 2h. Oh now the time limit has been changed to 2h 😁 and the scope is still the same. Of course, the projects are highly underestimated.

You know what? Top companies offer to travel after successfully passing the HR and technical screening interviews (1h30 total). I (and anyone) know where to focus for better use of my time.

Why Top companies don’t use take-home tests?

Top tech companies FAANG (Facebook, Amazon, Apple, Netflix, Google) and other respectable ones in Europe ( Zalando, Criteo … ), Asia (Alibaba …) DON’T give take-home assignments as a prerequisite for entering into the process. There is no shortage of candidates willing to work for them who would take the assignment in a heartbeat.

After passing the HR interview (30 min) and first technical screening (45 min -1h), the candidate is offered to travel on-site (flight+hotel+food). So why spend more than 4h only on the possibility to reach interviews? While top candidates invested 1h30 interview time and are compensated with travel and accommodation.

Take-home assignment is a huge opportunity COST. Qualified candidates will be turned off. For screening, a 1h interview is enough. Serious companies know that the better candidates won’t put up with crazy requirements like a lengthy take-home that most likely lands in black hope.

I once explained this situation to an HR lady: “We are a start-up, we don’t have the resources”. In admitting this, she hinted the compensation will be lower.

There are way more people applying there. The ratio of candidates to interviewers is higher. How do these companies find the time to interview candidates?

Look at their hiring funnel. The hiring funnel picture below is not at scale. First, notice the applicant pool is much larger and filtered for the HR screening. Second, the take-home challenge is replaced by a short technical screening.

Optimal hiring funnel

  • much larger applicant pool. At large companies, it is not uncommon that candidate profiles are filtered by machines ( ATS = application tracking software) based on keywords. To stand out from a large number of people, you should ask an employee (who knows you) to refer you.
  • HR: usual same friendly HR interview who will guide about the whole process. HR will prepare you for the materials and resources to study to ace the next technical/behavioral interviews.
  • Technical screening (45min to 1h): introduction of candidate and interviewer followed by one whiteboard coding, sometimes two (if you do well). If you performed one working solution that is not the optimal ( time and space complexity), you will get a 2nd technical screening. Depending on the companies, the success pass ranges from 5 to 20%.
  • interviews. 3 to 5 interviews based on algorithms, system design, and behavioral. A bit more difficult than screening. Generally, you are invited on-site, flight and hotel are offered by the company. In the Coronavirus period, the interviews are online.
  • the hiring committee will read the interviewer report. Depending on the headcount and availability, they will make an offer.

How to improve the hiring funnel for everyone

By improving, I mean to save time for everyone. Notice the major differences between the two funnels. What companies that send take-home assignments ought to do:

  • filter more aggressively the candidate pool. So recruiters will have more time to focus on quality rather than quantity. Some smaller companies have a drastic cut-off in screening. For example at 37signals, only the top 1-3% applicants are invited for take-home assignments after being selected based on their cover letter.
  • the screening interview is kept short. No BS take-home project for free.
  • if there is a take-home test, it MUST be short ( <2h), no ambiguity. Most take-home tests, whose authors haven’t even done their own challenge, are vague.
  • If you really require a long take-home test, do at least one of the following:
    • Create a draft of this assignment and make sure one of your engineers ( who didn’t contribute to the assignment) can finish this project under the time limit. Make sure the authors themselves can finish their own assignments under half of the time limit. Make their project reviewed by another interviewer in a blind test. Oh, and it will great if you send this project model to anyone who submitted for a correction.
    • provide an existing code base to avoid boilerplate code. Most day-to-day job are not greenfield project. A large portion is fixing and building on top of existing code. The burden on reviewers is reduced as well.
    •  zero feedback is pathetic. Give a constructive feedback rather than Yes or No.
    • or narrow down the top candidates and pay them. Think about how much an external recruiter costs ( retainer fee + placement fee ). A few hundred USD/EUR multiplied by the reduced pool of candidates (max 10 ) is really cheap. I even saw an assignment “Here are the UI screens, make pixel-perfect adaptive UI workflow”. It looks like some free work you could use later.Make it clear, you need the work to be done and pay talent accordingly. I just laughed at this recruiter 😉.
    • Bonus. Please record the live coding video to prove that you established a new Guinness World Record coding 😁 under your time constraint. And please send it to anyone who submitted your challenge.

Code challenge experiences that I liked

Now you might think I say NO to take-home assignments. There were several positive experiences made by smart people in a few companies:

  • One asked me which day I will be available to complete a 4h task: Sunday. The project manager sent at 9h the instructions, he was available to reply to anything while he was on a lake boat. By 12h30, I was finished with their Android app and sent it. They invited me for a day to Hamburg, they used this project as a base to complete new features. Discussion about design choices. Lunch with the team and job offered.
  • Live code challenge for 30 min on coderpad.io. Problem: you have articles in a cart, sort them by price then by quantity in descending order. You are allowed to use Google, but you must screen share. Me: “Oh I remember this sorting is done by a comparator, let’s google …”. 10 min later Congratulations, your solution works.

Practice

From now on, I prefer to stick to whiteboard coding style interviews because they are time-boxed and scalable. Once I understand a problem, I never forget and re-apply the same principles elsewhere. I understand people who are willing to train for a few months on LeetCode/HackerRank and refuse to spend one week on a take-home assignment.

It is fun to experiment with a project with the latest libraries ( still in dev, alpha, beta version, I am looking at Android Hilt, Jetpack Compose … ). But it is better to understand the CS fundamentals used to build these libraries or frameworks. The algorithms and data structure are timeless knowledge.

Excuses for take-home assignments

In the developer circle, I heard various stories: “I spent 8h on an Android tic tac toe project, they turned me down because they didn’t like the code formatting”

Personally, after submitting a clean project on time, I received this response from a company (one that I thought was respectful)

Thank you for taking the time and effort to complete the challenge. We really appreciate it.
Even though the hiring manager was impressed with your accomplishments, we have determined that another candidate is better suited for this role at this time. I am sorry to say we will not be moving forward with your application at this stage. Again, thank you for your time.
It was a pleasure getting to know your background better and I wish you all the best in your search.

What are your experiences on the hiring funnel for any companies that send take-home projects? Please comment below.

If you are on the hiring side, please post your funnel numbers (how many received tests /submitted/were invited for interviews)

One thought on “Companies that give Take-Home Tests: review your hiring funnel

  1. Any company who compliments or declines a take-home assignment without sufficient feedback should be named and shamed !

    I don’t interview for companies that do lengthy take home challenge for this reason.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.