The SDET Pendulum

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

At a recent internal forum, I hosted a panel discussion comprised of “senior” level testers at Microsoft. The panel was evenly split between managers and non-managers and we took random questions from the audience on career paths in test. The testers in the panel had experience ranging from 11 to 24 years in test. Some of the questions strayed slightly from the topic (e.g. “What does the future of test look like to you?”), but for the most part went well. The session was one hour, but could have held it’s own for at least another 30 minutes.

That’s pretty much it, so I suppose I can end this post here…

But one question came up that I want to dig into a bit, and I know that some of you who occasionally read this blog have some opinions on the subject, so fire them away please.

The question in mind may as well have been seeded by me, because it’s a subject I’ve brought up before. The question was: “Has Microsoft swung the pendulum too far in hiring only programmers as testers?” I can’t remember if my answer was “No, but Yes”, or “Yes, but No”, or “It depends”, but I’ll give a (much) longer version here for your benefit.

BTW – I would bet that if I asked most non-microsoftees the response they would give for this question, they’d say, “of course – my {cousin | dad | sister | friend } who is the best tester in the {company | city | country | world } can’t get a job there, and they’re {fantastic | awesome | superman }. They just can’t write code.”

And, in many cases, they may be right – but maybe not – or it depends. Some context is in order. I’m not the company expert on this, but my opinion and view of history is as good as any.

Our software is big and complex. We deliver platforms, operating systems, server products and put them into huge deployments. We need testers who can program to help solve the big testing problems that our systems present. Say, for example, you’re a tester on the Exchange team, and you need to verify that messages of varying lengths travel from one endpoint to another successfully. No problem – send mail to a test account and make sure it gets there… well…that solves one happy path. The reality is that the tester needs to send multiple mails through the system with a variety of headers and sizes to ensure that anytime the message is broken up that it still ends up at point B correctly. And, going through outlook is probably not the most efficient way to test this, so it’s probably better to construct the messages programmatically then send them through the system. Oh yeah – there are an almost infinite number of ways the server topology can be constructed, and those settings make a difference too (in other words, we need to deploy any arbitrary topology automatically on demand for these tests). As you can see, there are some obvious benefits to having someone who can program as a tester.

Microsoft has always had testers that can write code. Always. For much of our history, we’ve also employed testers who don’t write code (but who are still good testers). We’ve found over the years that our best testers and best test leaders had a programming or CS background. I’m guessing we also found that defining the career path (MS is really, really big on career paths) for testers was a bit easer to map out when we took programming background and application into account. We still use a lot of non-programmer testers in contract and vendor roles, and I’m guessing we’ll continue to do so. There are many great testers who have learned how to program, but over time we found (or I should say, my personal experience was) that when we fished in the pool of programmers for good testers, we found a much higher percentage of good people than when we fished in the pool of good testers looking for people who could write programs. So, for full-time “blue-badge” employees, we fish for testers where programmers live; we get what we want, and everyone is happy.

Hardly.

Note that in the above I referred to testers who can program. Where I fear we make the mistake in hiring is when we (or anyone else who hires “SDETs”) hire programmers to do testing. The difference is subtle, but important. A tester who can program is a tester first – but they are a tester who can rely on computer knowledge or programming skills to do their work better. A programmer who tests writes tools and automation first – hoping that it will help them be a better tester (hint – it doesn’t work very often). For the most part, Microsoft has testers who can program – and they are freakin’ awesome. The problems they solve and the way they go about it makes me proud, excited, happy, (and a bit jealous) all at once. It’s what SDETs should do.

The industry perception of the SDET appears to be a role of writing automation and tools all day. When I hear this, I think of all the great testers I work with and tell people “No, No – just because our testers can program doesn’t mean that’s all they do all day – they are testers first”. Then, whether it’s in 5 minutes or 5 days (but sometime soon after), I’ll see a blog or a tweet or some other sort of message from someone saying “I’m an SDET – I write automation and tools all day”, and I realize that there’s still some work to do. It doesn’t mean that most of our SDETs write tools and automation all day, but there are certainly pockets. My thoughts are that if you want to write tools and code all day, get out of my business – go be a developer. You’re not helping me (and you’re probably slowing me down), so leave me alone. I want to test, and I want to work along side people who feel the same way.

In the panel discussion I asked every hiring manager in the room to ensure that in the next interview they conducted that they looked for testers who used “technical” skills to solve their problems. Instead of making them write code on the whiteboard, give them hypothetical testing tasks and examine how and when they used the power of the computer to help them. For example, if I ask an interview candidate how he would know if a line drawn on the screen was straight and he started writing up an algorithm to scan the pixels on the screen and determine if the offsets were consistent with a “straight” line, I wouldn’t hire him (“Look at it”, or “use a straight edge while looking at it” are both acceptable answers). However, if I describe the message transport scenario I used a few paragraphs above and their answer is to ask everyone in the company to send a bunch of mail, I probably won’t hire you either. (I’m working on an internal presentation on hiring testers – I’ll come up with better examples :})

Every business is different, and your need for testers or “SDETs” will depend on your business. My call to action is this: If you wan to hire an SDET – hire a tester who can program – not the other way around”. You and your product will be much better off.

Share this:

Like this:

Related

11 Comments

Well said. Any tester that works for a large company knows exactly what you are talking about. There are those who test to become a developer and those who develop to become a better tester. I just wish that your view of SDETs was universal. I’ve seen a few associates join companies in this role and find out they are an automation slave.

That is very interesting post because made me remember about my Microsoft SDET interview. Well it was a couple of years ago and I think I have improved since then and hope to continue doing so.
I remember my interview was about testing a keyboard maybe or something similar, this part good I think. But the questions for programming part I am not comfortable with them even now. It was too abstract and too programming oriented and I think is bad. I think in testing you need to have courage to hire the right person instead of pleasing a developer friend or a higher manager.
You gave the example with sending email. “No problem – send mail to a test account and make sure it gets there… well…that solves one happy path. The reality is that the tester needs to send multiple mails through the system with a variety of headers and sizes to ensure that anytime the message is broken up that it still ends up at point B correctly”. Sending multiple emails through system or encoding them in certain way is easily possible in python or maybe there are open-source tools or a separate dev team can do it. For me python is a useful tool, I am not considering myself a developer by knowing how to use. I don’t think I would pass a dev interview for it also. But I would not spent days creating automated frameworks, no. I would just create simple scripts to help me achieve what I wanted. You don’t need someone who knows programming, scripting is enough. And a good tester needs to learn a little bit of that but not fill its priorities with it.
Windows is good btw for usability, gaming platform but everyone knows about the Windows blue screen. That’s why so many people use alternatives like linux and some people really hate Microsoft products.
I would say scripting may be needed like many other helpful skills for a tester but not give it 50% for it but more like 5%. Programming is a skill that is more controllable, more measurable and that’s why people tent to get comfort with it but it shouldn’t poison testing skills.
My opinion

Where I work, we are a mixture of both. Some testers get in from Client Support, fine at exploratory testing and with a good point of view about what our customers need and like. Others come straight from graduation, they are fine coding and automating, just need to know what the testing bug is about, some get it, some do not, and we all try to have fun at work. Somehow we all envy the others skils, I wish I could code, and my fellow automators would like to know better the app. What I think, is that there is never going to be any perfect rate, you just need to search a balance, and keep searching it as long as you and your company lives.

I totally agree with you in your viewpoint. I fit that description. I am a Tester who programs and uses it to do my job better.

This gets into one of my own personal viewpoints/opinion. There are basically two types of testers; Technical and Business. The Business Tester is a Subject Matter Expert of the business and how the functionality of the software is supposed to support it.

The Technical Tester is someone who has a technical view of the software. They have some basic idea of what the business need is, but they really dig into the software itself and the technology behind it. They know how to do things under the covers and by backdoors. They use technology and programming to solve the problem of getting a test done. Your Outlook / Exchange example is a perfect one.

I could go on, but this is definitely a discussion to have over a cold beer and a couple of hours.

If I am asked that interview question of whether the line is straight, my answer would depend on the context of the question.

Would someone who buys our product incur bodily harm if the line is not exactly straight, thus risking a lawsuit? If yes, then a more scientific approach to testing this would probably be my choice. If not, then you are right, using a straight edge while looking it should suffice. So my answer would be “It depends”. 🙂 I know it is a contrived example but just want to point out that my choice of testing methods are also dependent on other factor.

I agree with your straight line example, but would find it helpful if you could elaborate. The way you asked ” how he would know if a line drawn on the screen was straight?” does seem to imply a one time test. Now in a production application where that line could become unacceptable after any change to the code base and you would like feedback to the developers as soon as possible wouldn’t the person who proposed the algorithm be invaluable?

I am currently working on the hypothesis that the skills required to maximize data collection(SDETs) and the skills to find, diagnose, and assist in resolving as efficiently as possible(QA Engineers) are exclusive enough to warrant separating the roles. The former requires the skills to build up a solution and innovate. The latter requires deductive reasoning, and a high awareness of the product and customers. Of course I don’t think these roles are completely exclusive and overlap will assist. However, by putting people into the role that best fits them you can maximize their talents.

I find myself comparing SDETs to Biomedical Engineers coming up with the newest technology to detect a disease. Where as QA engineers are the Medical Doctors that need to analyze the patient and determine how to diagnose the patient and decide the best treatment.

hopefully it isn’t just supporting a cognitive bias (such as Confirmation Bias and Observational Selection Bias) that I might have with regard to testing.

Brandon – “However, by putting people into the role that best fits them you can maximize their talents.” yes! (an example (of many) a BA and a QA aren’t necessarily interchangeable)

David – I am not inclined to say that science and law see things the same way. Law seems to be more interested in following precedence.

Jokin – exploratory testing is useful, and an ongoing process. One consideration here would be time-boxing the effort since it is an open ended area. And I agree that some programmers don’t even seem to understand how to go pass confirmation bias testing.

Eusebiu – nice point about the distinction of scripting and programming

I’ve sort of grown into the testing role coming from tech support. I’ve also learnt how to program better after being inducted into the Software Engineering group. This article helps me understand how an interviewer would look at my skill set and certainly helps in writing a better resume.