The position is an entry-level position that consists of reading C++ code and identifying lines of code that are vulnerable to buffer overflows, out-of-bounds reads, uncontrolled format strings, and a bunch of other CWE's.

We don't expect the average candidate to be knowledgeable in the area of software security nor do we expect him or her to be an expert computer programmer; we just expect them to be able to read the code and correctly identify vulnerabilities.

I guess I could ask them the typical interview questions: reverse a string, print a list of prime numbers, etc, but I'm not sure that their ability to write code under pressure (or lack thereof) tells me anything about their ability to read code.

Should I instead focus on testing their knowledge of C++? Ask them if they understand what a pointer is and how bitwise operators work? My only concern about asking that kind of question is that I might unfairly weed out people who don't happen to have the knowledge but have the ability to acquire it. After all, it's not like they will be writing a single line of code, and it's not like we are looking only for people who already know C++, since we are willing to train the right candidate. (It is true that I could ask those questions only to those candidates who claim to know C++, but I'd like to give the same "test" to everyone.)

Should I just focus on trying to get an idea of their level of intelligence? In other words, should I get them to talk and pay attention to the way they articulate their thoughts, and so on?

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.
If this question can be reworded to fit the rules in the help center, please edit the question.

4 Answers
4

I feel a disturbance in The Force when I read "entry-level position" and "reviewing code for security vulnerabilities" in close proximity to each other. Seriously. I've been grinding out code for 40 years and I still can screw up on stuff like this. The people doing code review for security problems need to be more experienced than the people who wrote the code in the first place.

You're far more likely to find actual buffer overflows and sanity-check failures by doing things like Fuzz testing than you are by hiring "entry-level position" programmers. This is the type of tool that actual Bad Guys® use in trying to crack an app. (BTW, a good round of Fuzz testing can prove very embarrassing to the original coders.)

I think good interview questions would be:

Strictly off the record: how many sites have you broken into?

How old were you when you first hacked into someone else's site?

Have you served any jail time for illegal access to a computer system?

When I interview candidates, I will sometimes run my own version of the word association test that has been used by psychologists since Karl Jung.

In this test, I have a sheet of paper that lists twelve to twenty common terms and concepts from a topic area like embedded systems, testing, or for your example, software security / writing secure code. Typically, I have two copies of this list, one for myself which I mark, and one for the candidate. I request the candidate spend no more that 10-20 seconds on each because I am looking for breadth rather than depth. Once we complete the list, I can ask for more depth. I also have the candidate return the list.

My list for software security might include:

Basic

buffer overflow

code injection

SQL injection

strcpy();

format string vulnerability

authentication

denial of service

password strength

cryptographic hash

certificate

SSL

sand boxing

firewall

security risk model

security by obscurity

salt

protected execution

Medium

AES

non-repudiability

PKI

web of trust

digital signing

elevation of privilege

acl

group policy

DMZ

man in the middle

WEP

default permissions

TPM

smart card

NFC

CAPTCHA

challenge / response protocol

replay attack

Expert?

Schnierer

Alice and Bob

Malory

Tempest

digital cash

touch screen voting machine

Enigma

cipher

Certainly these lists need some tuning. Interviewers should take precautions to insure that the questions they ask are relevant, so you might want to evaluate each item against your business, but if you do web sites, I think you need SSL, SQL injection, certificate. If you ever use passwords, you probably need to understand what security by obscurity means and what it implies you should never do. If your team administers servers, firewalls, salt, authentication, acl, and probably group policy are important.

Security threats are growing more common and more nuanced. If you can get someone who knows a lot about software security, grab them. If you can't, try to talk someone in your team into becoming an expert.

Don't ask questions, give them some code and ask them to find the vulnerabilities. Then ask them to prove to you each bug or vulnerability. Make sure you have an answer sheet with the correct results and reasons so that you can show them it was a fair test.

I did this kind of work for a while and when people had to stand behind their work in front of the paying customer, you would not believe how fast some of them would not stand behind their work and pass the buck. It is one thing to say you found a bug in a report; it is another to convince someone in person who probably has much more knowledge about the code and probably more years of experience than you and explain to them where they made a mistake.

More importantly, why are you not using tools to do this? Granted the tools are not 100% full proof but are faster and if you get the same result from two different tools you can be pretty sure it is a true problem. Also since you are paying a person to do all of the code manually, the tools can build a working list of possible bug candidates that a person can check.