Companies that give Take-Home Tests: review your hiring funnel

TDLR: This is my opinion.

To candidates: Go where you’re treated the best.
Why would you spend X hours upfront on an unpaid project to be interviewed?
While at top tier companies, if you succeed the 1h phone interview, you’ll be invited on-site with travel costs paid.

To companies: DON’T SEND “full application to complete” take home challenges to filter out. It sends the signal: “we are OK to waste hours of your time, so you can have the privilege to interview”.
Instead, use a short interview screen. 30 min is more than enough and what a proper code review would have taken.
The quality of people will rise.

To be a known quantity, please share your experience in this survey . A list of good companies will be published.

Take-home projects are heavily in favour of company side: no time invested, free consulting.

Some argue take-home challenges reproduce the real work environment with developer tools that anyone can do at their own convenience. The very same camp argues that whiteboard coding (algorithms + data structure) does not reflect daily routine. However the take-home approach has a huge time investment and the risk to be ghosted (ignored without feedback) is high.

The reasons why take-home assignments became popular is time invested asymmetry. It takes much less time to review than to do an assignment.
It basically sends the message: “we prefer to waste 100 hours of candidate time than a few hours of company ‘s time”

However take-home challenges themselves as a filter is just a wrong strategy.

Their hiring funnel looks like this

Here are the steps in the hiring funnel:

  • Applicant pool. The candidates who applied are in the applicant pool. The resumes and cover letter are usually read by a human in smaller companies who may call each candidate for the 1st interview.
  • HR: Major HR interview is a friendly 30 minutes chat. After everyone will receive a coding interview to complete at your home on your time. Some passive developers are contacted on various channels ( Linkedin, StackOverflow … ) by the HR/recruiter who try to bring attention.
  • Most recruiters have no way to filter technical people, usually almost everyone who interviewed with the recruiter will receive the coding challenge.
  • take-home challenge: Once the homework read, the majority of developers believe the challenge is not worth their time and will stop there. Among the candidates who interviewed first with HR and received the take-home test, how many of them complete the task? Few recruiters were transparent with me about the percentage of people who submit their code. My guesstimate is fewer than a third submitted for an established product company . As Gayle mentioned if this is screening round, it should be kept short.
  • interviews Often the code challenge is not even discussed, it was only used as a filter. On several occasions, I like that these interviews served as an open discussion for design choices and the take-home project serves as a template to add additional features during a live coding interview (googling is allowed)
  • hiring committee (if there is any). Interview reports are read and the salary range is discussed. An offer might match.

Pain points in this funnel

You can already see two major pain points in the funnel :

  • Almost everyone in the applicant pool was interviewed individually by HR. HR has to sell candidates to complete the homework. The HR is really busy to filter, interview, follow up with candidates. The focus is on quantity rather than quality.
  • survivorship bias on who completed the challenge. The hiring managers may believe that only successful candidates were the ones who submitted and may brush off that those who didn’t submit not were up to the challenges. In their vision, the take-home challenge is a test of worthiness for the next interviews.
  • Do you really think qualified candidates will do the test? People in demand will simply prioritise better alternatives.
  • very few candidates submit the take-home test. Because full-time employees who apply elsewhere prefer low hanging fruits. Didn’t you notice the HR keeps sending reminders about the project status?

If the conversion rate is low, something is wrong in the funnel. Which leads to why don’t candidates just submit the homework?

Take-home challenge is not scalable

The take-home test approach is not scalable for both the interviewers and interviewees. As each project is unique, the interviewers have to reply to candidates who request clarity (if the authors ever reply at all). Reviewing the submitted project takes time to understand the logic. The reviewer may spend more than 1 hour to review correctly. This hour should be rather spent on tech screening.

At your work, a full code review of feature request may take more than one hour. Yet the scope is smaller than a full blown project.

Project scope & time underestimated

I don’t mind to complete an automated graded test online for 1-2h.

However, most take-home tests are the type “Build an app/website that X and Y from scratch. We expect tests, threading, SOLID architecture, deployment, production quality, offline mode, 100% test coverage … You have 4h (to unlimited time) to complete the project”.

Were the expectations and scope of the test stated properly? Often you have to navigate with ambiguity and the project scope is ridiculously too large.

I said I don’t mind to spend 2h. Oh now the company set the time limit to 2h ūüėĀ and the scope is still the same. Of course, the projects are highly underestimated.

For tech screening, online tests are much faster to evaluate candidates.

Automated tests as tech screening

Since the code challenge have to be reviewed by humans. It is much better to filter with online automated test (coderpad.io ) . Main advantages are:
partial grading. Expectations are known. Different reviewers have different way to grading.
time efficiency for the company. No one has to correct behind the scenes. Candidate

No feedback, no review at all

With take-home assignments, rejection without feedback is common.
While if I do an in-person interview, I guess where I missed the answers. The HR always follows up with feedbacks.

You don’t know on what criteria you will be evaluated. Your application could be rejected on “unstated” trivia criteria:

– “we don’t care about problem solving, we only care about your code style.”
– use the wrong formatting tool. In the developer circle, I heard various stories: “I spent 8h on an Android tic tac toe project, they turned me down because they didn’t like the code formatting”
– use a popular library that is not known by the team
– use an architecture that the team is not familiar with
– deliver a high quality project, over-engineered for the reviewer
– …

Intellectual property

Imagine, you gave your best shoot and product of your work. You submit. No feedback, you are rejected.

However your code solved a business relevant to this company. You solve in a novel approach. They use your code in production.
You are still owner of your code, but you were not paid.

Only complete a challenge that a company has already solved in different ways long ago.

Why Top Tiers companies don’t use take-home tests?

Top tech companies FAANG (Facebook, Amazon, Apple, Netflix, Google) and other respectable ones in Europe ( Spotify, Zalando, Criteo … ), Asia (Alibaba …) DON’T give take-home assignments as a filter. There is not even a take-home coding challenge. There is no shortage of candidates willing to work for them who would take the unpaid assignment in a heartbeat.

After passing the HR interview (30 min) and first technical screening (45 min to 1h), the candidate is offered to travel on-site (flight+hotel+food). So why spend more than 4h+ only on the possibility to reach interviews? While top candidates invested 1h30 interview time and are compensated with travel and accommodation.

Take-home assignment is a huge opportunity COST. Qualified candidates will be turned off. For screening, a 1h interview is enough. Serious companies know that the better candidates won’t put up with crazy requirements like a lengthy take-home that most likely lands in black hope.

I once explained this situation to an HR lady: “We are a start-up, we don’t have the resources”. In admitting this, she hinted the compensation will be lower.

There are way more people applying there. The ratio of candidates to interviewers is higher. How do these companies find the time to interview candidates?

Look at their hiring funnel. The hiring funnel picture below is not at scale. First, notice the applicant pool is much larger and filtered for the HR screening. Second, the take-home challenge is replaced by a short technical screening.

Optimal hiring funnel

  • much larger applicant pool. At large companies, it is not uncommon that candidate profiles are filtered by machines ( ATS = application tracking software) based on keywords. To stand out from a large number of people, you should ask an employee (who knows you) to refer you.
  • HR: usual same friendly HR interview who will guide about the whole process. HR will prepare you for the materials and resources to study to ace the next technical/behavioral interviews.
  • Technical screening (45min to 1h): introduction of candidate and interviewer followed by one whiteboard coding, sometimes two (if you do well). If you performed one working solution that is not the optimal ( time and space complexity), you will get a 2nd technical screening. Depending on the companies, the success pass ranges from 5 to 20%.
  • interviews. 3 to 5 interviews based on algorithms, system design, and behavioral. A bit more difficult than screening. Generally, you are invited on-site, flight and hotel are offered by the company. In the Coronavirus period, the interviews are online.
  • the hiring committee will read the interviewer report. Depending on the headcount and availability, they will make an offer.

Hire the best, not the desperate

The best job candidates have choices and will not ignore those jump through hoops processes.
The desperate have time on their hands to complete the challenges. I would advise to work on personal demonstrable projects is better use of their time.

Whiteboard practice

From now on, I prefer to stick to whiteboard coding style interviews because they are time-boxed and scalable. Once I understand a problem, I never forget and re-apply the same principles elsewhere. The Leetcode efforts are amortized. If you believe that Leetcode is learning by heart, you are doing wrong. You need to understand the problem solving approach under the hood.

On whiteboard coding , the metrics are clear: the solution passes all tests, thought processes, efficiency in time and space complexity.

It is fun to experiment with a project with the latest libraries ( still in dev, alpha, beta version, I am looking at Android Hilt, Jetpack Compose … ). But it is better to understand the CS fundamentals used to build these libraries or frameworks. The algorithms and data structure are timeless knowledge.

Usually, employees working at companies without whiteboard interview might be good technician of a particular framework and know by heart the API methods. But they lack basic data structures (Map, Set) and CS fundamentals. Being on the hiring side, I interviewed such candidates coming from these companies.

Their solutions on the whiteboard are far from the most optimised, almost no notion of Big O notation. To the question: “Propose a solution to find an element in an array in constant time”. The candidates would use the built-in binarySearch in Java on an unsorted array. Instead of pre-caching the values of the array in a Map or Set. My colleagues unanimously said: “For sure, NO hire.”

What are your experiences on the hiring funnel for any companies that send take-home projects? Please comment below.

If you are on the hiring side, please post your funnel numbers (how many received tests /submitted/were invited for interviews)

PS: from others experience, companies with take-home assignment are only “interviewing but not actually hiring”.

Views from other industries

I once spoke to a friend who is actor, David . Each casting is a free interview. But the recording won’t be used for commercial purposes.

No actors, dancers will record a commercial scene for free. What you told me is :
Imagine you go to as many restaurants and tell them:
I will pay only the best after I sample each potential restaurants for free.”
Won’t you feel bad to about this to each restaurant? This is exactly what these companies are doing to you.

Dear fellow engineers, don’t get scammed, you are in demand. You have the power to say NO.

FREE WORK?

Ask other professionals for free sample. Try to ask a a locksmith to open your door for free? It takes a few minutes of their time.

Now developers are asked to spend hours on a single take home assignment. Do you think to it’s fair offer your time for free?

A few times per month, I receive a cold phone call from a recruiter. “We have this amazing opportunity. Would you like to hear more? … We ask for a small code challenge, it will take only 3 hours”. After a quick glance, the project is a full-fledged app with pixel perfect Sketch designs. The assignment asked for unit tests, integration tests, complete CI/CD pipeline, [docker & kubernetes]. It would take at least a week to complete. My response: “Listen , you are the ONE who contacted me. I never applied.”
And your response should be:

Hi [recruter],
To be perfectly honest, I'm really excited about [company] and got a great impression of the company from my interview with [you or engineer]. I estimate this assignment will take at least H hours of work. My contract rate is ‚ā¨$ X/hour. I can complete this assignment for [total] and return it to you completed by the end of next week. If that isn't acceptable, I do have an extensive portfolio of my work available on github (link). Please let me know how you would like to move forward.
Best,

You could probably charge for 165 USD/h for 20 like this guy did.

Even Free work from reputable Firms?

Reputable firms are classical banks (not online banks), retail stores or any companies with 9 digits annual revenues.

I submitted code to a firm. The reviewers were impressed but the firm wanted more features, it would take 2-3x times. I politely declined. The manager invited me for the rest of interviews. So the code reviewer was not even important enough to be part of the interview.

Another time, I sent the project. The company sent feedbacks, but the position was refilled by an internal employee.

Or the best, I inserted analytics in the project and readme.html ( not .md file), so that when the documentation or project are opened, I’d be informed. The company never opened and sent a blank template “no pass”.

Most likely, you will gain nothing from completing take home assignment except adding to your public portfolio. My repeated and peers experience is that simply being asked to do a “take home project” is a strong indication that companies are cheating and not actually hiring.

Survey

Please share your experience in this survey

The goal is not to shame and name the authors. But to get an overview in the hiring funnel and share the fair players in this coding homework game.

 

Google interview guerilla : my story

For most people the name Google is a verb for “search on internet” or there is now a saying ” Always use Google before you open your mouth”. For most software engineers , it remains the king of the hill in hi-tech. Most of my knowledge reading come from Google engineers. I once encountered Romain Guy, an engineer on the Android team. Then later I discovered stackoverflow.com a Q&A exchange for programmers, where Jon Skeet is a living legend. The guy spends 20% of his work time on Microsoft technologies, and he knows way better than the team who designed them at Microsoft. Personally for me, Google is by far the most selective of all tech companies. It’s the McKinsey of the high tech.

Last year around November 2010 , a technical recruiter from Google contacted me, he was scouting on any networks . At least I looked good on the paper. But what about the real stuff ? I did not respond because I heard rumors about their interview and had other priorities.

But in July 2011, I saw a job update that may interest me. One was from Google ¬†Paris, where I’m living. As far as I know , a software engineer position at Google in Paris is rare they only have marketing. (though¬†today¬†they just had a new office opening ) .¬†Grabbed myself ¬†” You’re going to regret it , Ray for the next 10 years if you let this occasion slip” . Some friends say it’s just a company, no myth behind.

Before you send a resume

I got many job offers previously without reading a how to write your resume, preparation at a job interview. But still for me Google is another beast. I purchased “the google resume” by Gayle Laakmann McDowell. In fact I learnt some truths : when you write about past experiences put quantifiable accomplishments . Had I known before, I’d have done it for previous positions.

This post is also interesting in this regard http://www.joelonsoftware.com/articles/GuerrillaInterviewing3.html

Interview 1 phone screen

Well I admit, I almost did nothing during the day the resume was sent and the first contact from Google staff: 2 weeks passed. I should have studied. Someone from Zurich arranged for time schedule on 20 minutes phone interview. We agreed on ¬†November 4 Friday “In Paris, your resume was turned down. I’m a recruiter in EMEA, based in Zurich”. Turned down Paris, the place I applied to. Zurich it’s even better . I saw the cost of living¬†¬† , the pay is more than double and cost of living is not as high. That’s worth it.

She said she saw on my resume “automatic testing” . They have open positions on the Chrome team for automatic¬†test . Google Zurich is the biggest office and the main engineering force in Europe. Not what I was expecting. Rather than saying “yes” to enter the boat and hope to land later on the dream job. I stayed true to myself :
“As great as Google is, I’m not really interested in Chrome. Don’t you have exciting projects on mobile Android, Maps .. ?”
– Don’t worry . If you want I can forward your CV to London, it’s the main mobile expertise in Europe.
Yes great. She answered several questions and presented the company in general.

Interview 2

After the weekend I received an email from London.

With her agenda, we agree for an phone interview Thrusday 10 November.  I booked a room conference at my work place. Day d , just a few minutes before some dumbass take my room.
– Did you check on Outlook the planning ?
– yeah but we need it now . So who is on the phone ?
– it’s a girl. ( white lie ;¬†can’t say it ‘s for a job interview )
– well you don’t have the priority.

Don’t be afraid to postpone for your phone call. The recruiter will understand it. Luckily November 11 is a day off in France . We chatted what interest me and the open positions.
– So we have a smallish mobile team for Youtube. with her Eastern European accent . ¬†Because technology changes fast , we are looking for people who can adapt fast and have a strong core understanding. You will write on google docs. The code you write must be compilable.¬†Are you comfortable with Big O notation, algorithms and data structure ? ¬†You’ll get these at the technical interview.
– Oh actually the recruiting is so fast. I wanted to buy more time to study. I practiced on “Cracking the coding interview” by Gayle Laakmann.
– OOOHHH so you already prepared yourself. Do you know people at Google ?
– Not personally , Jon Skeet answered some of my questions. And I bought Reto Meier ‘s book . I mentioned both because they’re working in London so she may know them.
– AH GOOD, I will give you some resources where you can prepare for the interview.
An hour later, she sent a list of topics the engineer might ask.

Technical warm up

For this one, I took 2 weeks of revision. Most topics are common place in Computer Science program : singly/doubly linked list , sorting and searching …
Others I maybe knew : trees & graphs , red-black tree, ¬†stacks & queues … Others I never heard of such as¬†NP-complete¬†.

Or the infamous “estimate how many ¬†golf balls can fit inside a bus school ?” . Not hard once you know how to deal with Fermi problem .
I practiced the problems for a few hours before sleeping. By practice I mean : coded in developer tool & tested it , wrote the same code on paper, wrote on google docs.
The syntax highlighting , code completion make life easier . It’s like comparing write Chinese with pinyin on the keyboard, and handwrite the Chinese characters on the whiteboard.

I didn’t have ¬†to sign¬†any NDA . I was emailing some friends about the content of google interview with gmail . Hope big brother is not watching ūüôā , they don’t care .

Interview 3 : real stuff

The day Monday 28 November at 10 am UK time. The engineer was late by at least half an hour.
– So I want you to check whether a #@## puzzle is correct or not ?
– what kind of puzzle ?
Р #@## puzzle .
– Can’t you just write your question ?

I prefer the Eastern European accent than the thick native English speaker accent . He sent me a link . Oh I understood : a sudoku puzzle. I barely know the rules. I asked:
РSo what are the constraints ? Each number from 1 to 9  is unique on each row and column . Other contraints ?
He gave me a hint each 3*3 small grid contains a unique number from 1 to 9.

Obviously the grid is a two-dimensional array of integers. I wrote the method to check uniqueness on the row. Idem on the column.
No impression of conversation, I was stressed to have silence during my writing.

At the end , he asked what I didn’t do . I answered :
– Check the extreme case values : what the puzzle contains negative numbers or greater than 9.
– check the size of the puzzle .
He seemed satisfied to hear these answers.

After this, in retrospect I found more efficient & shorter ways to solve the problem. Though I don’t think I can do it in 30 minutes .
But in the privacy I could do it much more efficiently. Not my lucky day.

45 minutes is short. But if you think what about you can accomplish during this time frame.
In the afternoon, I went to work. My manager scheduled a meeting at 5pm to know my work progress, I had 3 hours of work so 4*45 minutes. I was as productive as having worked 8 hours this day.

Next

A week after, the london recruiter contacted me. On the phone she said : ” Sorry about the delay, the software engineer was … ¬†But we decided not to go further .¬†We can‚Äôt thank you enough for your interest in Google‚Äôs careers and unique culture; we hope you will remain enthusiastic about our company.”

Anyway I would not like to work Mr Sudoku , not my fit . But I feel energized by the smartness of engineering at Google. You really feel the engineer driven  culture. But still felt it was more about luck.
You’re tested on a small set of knowledge . It’s more about to have encountered the type problem before you’re asked. Brain teaser like : “Why are man holes round ?” , if you read it once, you know the answer . Otherwise it’s almost impossible to answer it or it takes much longer.

No creativity tested, they don’t value design skills . Steve Jobs would not get even a job at Google¬†¬†if he passed the standard interview . Come on, his inventions are leading the industry. His presentations make fans excited. Does he know Cocoa API¬† and Big O notation ? No.

I have the feeling they don’t care much about what you did previously. The sudoku guy admitted he didn’t read my resume. ¬†It leaves many holes in the recruiting process .¬†It’s more relevant to test a candidate on the field he/she will be working on. Ask a marketing person on sorting algorithms tells a lot about the ability to perform, huh ? ¬†Anyway I’m glad I went through the interview. At least I know how to prepare for the next interviews either Google or others.

And to demystify their eliteness, the folk who speak at conferences & publish books only represent a very small minority .

Wisdom

In working environnement no one needs to know about the fancy algorithms . In real life, most IT¬†professionals have a buzzword ¬†of keywords in their resume . But very few know how to implement themselves. It’s just magical. And the truth some bozo have these words in conversation and CVs, but never applied them. They’re the kind of people who talk about cloud computing because they have iCloud devices but never understand the most basic of distributed systems.

And worst in the entreprise IT world , managers have the reasoning the outcome of project is proportional to the number of people involved. The more people you have, the more things can be done. Especially on consulting side where more people equals charge the customer more  ==  more money.

Quote People who Get Things Done but are not Smart will do stupid things, seemingly without thinking about them, and somebody else will have to come clean up their mess later. This makes them net liabilities to the company because not only do they fail to contribute, but they soak up good people’s time.

Writing code is not just interpret the requirements, but the talent to implement and ship a product. There are some fields where an individual can accomplish more than a group of people. Take football for example , anyone can kick the ball , a few can score a goal, much fewer can score against pro players . Lionel Messi , one the top football player, will score better than ten random people picked in the street. So the number of players is more important than individual talent ?

Anyway I feel better to invest time in learning the fundamentals because they still stay true over time . While technologies get depreciated pretty quickly, 1-2 years . Android 1 came out in 2008. Android 4.0 was released in 2011. A lot is not backward compatible, a lot new features arrived.

At least I have six months ¬†to prepare ( if I don’t have a better opportunity before ) , instead of 2 weeks . I hope you enjoy this reading. Feel free to reply.