Technical Tests

Sam’s Note: This is a guest post by one of the smartest people I know, the amazing Jim. It’s a brilliant read and I hope you enjoy it as much as I did.

Jim Bennett is an international C# geek who has spent time hanging around in finance and startups for the past 15 years in 3 different continents. When he’s not writing desktop WPF apps for large banking clients he’s building mobile apps using Xamarin, blogging and being amazed that his 2 year old is better on his iPad than he is.

He doesn’t like pina coladas or walks in the rain, but does like beer and thai food and will happily talk tech whilst you buy him either (or preferably both).

Let me start by saying I LOVE technical tests. To me they are the most important weapon in the interviewers arsenal. Asking text book style questions or scribbling on a whiteboard is all well and good but when you are employing a coder you really need to see how well they code. It’s no different to employing a wedding photographer or an architect – you want to see how well they can actually do the job by looking at their work before you commit.

So what do I mean by a technical test?

What I am referring to here is giving the candidate a problem that they can code in their own time and their own style. It should be in the main programming language that you need from them (or a choice if you are employing for a polyglot or trainee role) and should be related to the domain you are working in. The instructions to them should be simple and concise but otherwise they should be left to solve the problem their own way. Remember, the aim here is to allow the candidate to show what they can do, give them a chance to shine.

Before I give an example, I must start by saying that my background is not in Java. I’ve been a C# developer for many years mainly doing front end development for the finance industry, so the bulk of the tests I have created in the past have all been UI based. The example I will be using is from a C#/WPF technical test I would give to front end application developers, but the basic principles can be applied to any test.

The test I have given in the past is to build an order blotter. This is a data grid based application that shows rows of data that correspond to orders from clients to buy or sell shares. These blotters would show the symbol for the shares being traded (e.g. MSFT for Microsoft), if they are to be bought or sold, the amount to buy or sell and the current market prices. The order information would update only when the shares are actually traded, but the prices update regularly – sometimes many times a second.

The candidate is provided with a library that publishes order information with constantly changing prices and is asked to create this blotter application to show these orders and provide live updates of the prices. They are also asked to support submission of an order through the given library by creating a simple dialog to enter the order information. The candidate is asked to support unit testing and ensure the application is responsive. These last two statements are left deliberately vague as the candidate should be aware of the standard patterns and practices that would be used to support these. Other than that they are free to code how they see fit.

As an interviewer what I am looking for in their solution is:

Good, clean, well structured code.

Use of standard design patterns – especially good separation of concerns to allow unit testing

Appropriate naming of classes, methods and properties to provide self-documenting code as far as possible.

Something that works – delivery is one of the most important features of software so the app needs to work just by getting the code, building it and running it.

A reasonable looking UI. In the timescales given there isn’t time for full user experience analysis but there should be basic attention to detail like labels, input controls and buttons lining up.

I also like to add a small gotcha. In this test my gotcha was a 5 second pause when submitting an order. As a UI based test responsiveness is an important feature. If you make the call from the UI thread the app will lock for 5 seconds which is a very bad thing. Any programmer who doesn’t care about this lock up or, even worse, doesn’t notice is not a good UI programmer and is an instant fail.

How should I go about creating a test?

The ideal test should be designed to find the kind of developer you need to do the job that you are employing for. If you are looking for a web developer then it should obviously be web based, if you are looking for a front end developer then it should be UI based to show both coding and UI/UX skills, for a real time code developer it should be something with a performance consideration. You could even do the test given in the example above as a command line app in Java to test back end developers. It should also be simple to explain, with you providing everything needed except the computer and development environment.

It should also be short. You are asking a candidate to fit a few hours into their schedule around their current job, family and other commitments. Your timescales for them to deliver their solution should also be long enough to give them time to complete without feeling rushed. If you give them an 8 hour test and 1 day to do it in when they are at work then don’t be surprised if no-one wants to do it. I find a 2-3 hour test with a week to do it (allowing for a whole weekend, with more time for existing commitments such as a holiday) is about right.

Most importantly, you should do the test yourself. So should other members of your team. This allows you to ensure the instructions are clear, everything is provided and the timescale is sufficient. If it takes you 8 hours to deliver a solution you can’t expect a candidate to do it in 2.

You should also be available to answer questions on the test once the candidate is working on it. Not dumb questions that show that the candidate is not up to scratch, but if they ask for a clarification on a requirement then you should provide answers – especially if their first language is not the one that you’ve used to write the requirements.

Assessing the test

You’ve designed the test, made sure it’s doable in the timescale given and sent it to your first candidate. A week later you get the solution back. How do you then asses it?

The first and simplest assessment is: does it work? Getting the test back to you can be hard – if any required binaries are included they might get blocked by corporate email scanners so you have to be very flexible about receiving the code. One of the best ways is to use a private Github repo that you share with the candidate (assuming your company allows Github access, some are very strict). But once you have all the code you should be able to do a one click build and run. Anything more complicated (such as updating packages from a public repo) is fine as long as the candidate has documented this and it seems logical. If it doesn’t build, doesn’t run or compiles with too many unacceptable warnings then it’s an instant fail. I know this sounds hard but if the candidate can’t create a working simple program then this is a big red flag. This is the chance for a candidate to sell themselves by showing what they can do – if they can’t so something that works in a situation this important then it’s unlikely they would be bothered to do a good job when they are actually hired. I know of at least one manager who ignored a test that didn’t build and was badly coded and was talked into employing the candidate anyway by a recruitment agent – and the candidate turned out to be a really bad developer.

The second thing I look for is this: does it fulfil all the requirements? Being able to take requirements and convert them to code is an important part of software development, so their program must fulfil every requirement. You will need to be flexible on interpretation. As mentioned above you should be available for questions but in some cases candidates don’t like asking questions or don’t think that they can ask. This means that they may interpret something a different way. Any major requirement misses then I fail the candidate.

Once I am happy with the delivery side, I then look at the code starting with the class or file structure – is it logical and well named? I then dig into the code. Does it use the right design patterns, is it easy to read (allowing for language issues), is it testable and does it have tests? Essentially, would I allow this code into my repository and release it to production. Remember though that this is only a short test so you have to make some allowances such as only having a small amount of test coverage to illustrate that the code is testable rather than spending time doing full coverage. You also need to be flexible with style – for example I hate one true brace (putting the opening brace on the same line as the for loop/if statement etc.) but would never fail a candidate for using it.

If the code is all good then I would invite the candidate for a short interview, either in person or over the phone to discuss the test. What I would be looking for here is for the candidate to prove they actually wrote the code instead of outsourcing it (yes, it does happen). This would be a simple discussion, get them to explain the code structure, what code does what requirement, discuss the patterns used and demo the functionality including how it matches the requirements. Some places insist they do the test on-site to ensure they write the code themselves, but I prefer at home with a follow up discussion. On site means a number of hours out of their work day and if a candidate is interviewing at a number of places they may not have enough holiday for all the tests. On site also can make candidates nervous, exam style nerves can impact their performance and not let them show their best work. On site also means using a company provided computer which may have different developer tools to what the candidate is used to. This can have a negative impact on productivity which becomes a major problem when you only have a couple of hours to do the test. For example as a C# programmer I am used to Resharper so if I had Visual Studio without it I would be like a fish out of water and this would impact my productivity. This would be the same for Java programmers used to IntelliJ who had to use Eclipse.

If the code is bad I would provide constructive feedback to the candidate along with the rejection. Everyone should be given the chance to learn and grow as a programmer and if there is any way we can help weaker programmers get better then we should do it.

How does this test fit into the interview process?

I like to do this test at the very start of the interview process. My normal flow is to get a stack of CV’s via recruiters or adverts, wade through them looking for relevant experience (and ignoring their style, some people just can’t write CV’s but are great programmers). If they tick all the boxes for what I am looking for I send them the test. If they refuse to do the test then it’s into the bin with their CV. If they don’t have the resources to do the test (such as not having a computer) then again into the bin – software developers who do it as a job with no passion outside of work are not in my experience the best performers.

Once I get their test solution back and do the follow up chat then it’s on to the rest of the interview process. At this point you already know they can code so now it’s the time to look for cultural fit (do they get on with the team and you), sell them the job and make sure you can all work together. Some technical follow up is good to understand the depth and breadth of their knowledge so you know what training is needed or can work out a plan to get them up to speed. It also allows you to ensure they have the basics covered. If they ace the test then I don’t think you need to hammer them in too much detail – just enough to ensure the test wasn’t too easy.

The last stage of the technical test – feedback

I like to ask all candidates, whether they pass or fail, for feedback on the test. It should be a continually improving process – if everyone fails then maybe your test is poor. If everyone passes then goes on to fail at later stages then maybe it’s too easy. If it takes to long then it needs to be shortened. Your recruitment processes should always be improving so getting this feedback and actioning it is vital to being a good recruiter.

The other side of the fence…

As a candidate, I also love technical tests. They can be a fun challenge, allow me to show off what I know and can do, but most importantly give me some confidence in the team I will be working with. As an employee I want to work with smart people, people I can learn from and can help me grow as a programmer. As part of my interview process as a candidate I also need to interview the company to ensure they can provide this kind of working environment, and an interview process that doesn’t filter out bad developers by verifying their coding ability is a warning sign that the existing employees might not be that good.

Summary

In summary – technical tests rock. If you don’t have one then hurry up and create one before you recruit anyone else. And prepare it now. You might not be recruiting currently but it takes a few days to get a good test ready so you need it done before that stack of CV’s lands on your desk and you boss demands you hire someone yesterday.

Share this:

5 Comments

It’s great if you can get people to do this.

I suspect you live in an area where jobs are rare, and people only apply for a small number of them. If you talk to half a dozen, this becomes a notable burden. I’d a least suggest moving this later in the process, after a phone screen, at least.

It’s very easy to say “If they refuse to do the test then it’s into the bin with their CV”, but someone who is being actively recruited is not going to stop for your test. This filters out both the top, and the bottom end.

It’s an interesting point and something that at my firm we’ve batted back and forth for sometime. The “homework” as we call it is great for weeding rubbish people out quickly who can’t code, but it obviously runs the risk that good candidates just won’t do it. On the other side of the argument, this attracts some candidates as you’re instantly saying “we find good coders”.

I personally learnt a lot from Jim’s post. We never gave feedback for candidates because, as you say, it’s burdensome when you have a lot of these (and we have a lot of people applying; I’ve used homeworks in Hong Kong, Jim has used them in London and Hong Kong so a lot of potential applicants) but I now think next time I would spend the time to do this. But, I would have to put a phone screen in before this point because it would become untenable otherwise.

I had technical phone interview first; I passed the interview. Then I had an on-line test, I passed. They offered a face-to-face interview, I went for the interview face-to-face. During the f2f interview they told me that they wanted some one can hit the road and run. I asked them if they care how I run; they said they did not care as long as I got the job done. I got the job offer, but I did not take it. The recruiter was not very happy with me; I apologized, and I gave them my reason. Not once I was asked about my integrity, not once I was asked if I could think critically……. So sorry if my comment upset readers, but that is how I feel.

Hi Andrew, I say good for you. One of the key things I feel people don’t understand is that when you’re being interviewing for a role you are also interviewing the company for it’s suitability for you. If a company asks me stupid questions through the process and clearly does not value what I value in a role then it’s irrelevant of whether I get offered or not, I simply don’t want the job.

[…] Bennett and talk about technical tests. I loved his article he wrote for CJIQ (you can see the post here) which I took a ton of value from and really wanted to sit down with him and expand the discussion. […]