The Testing Show: What Makes a Tester a Tester?

October 26, 2016

Do you have to be a career tester to perform the testing role? If you are facilitating, coaching or leading others, are you testing? Does it really matter who does the role, as long as somebody does it? The Testing Show is back in the studio and chatting with Qualitest’s Yaron Kottler on exactly these weighty questions. Needless to say, we discussed these ideas of “what makes a tester a tester” and then some. Also, the panel shares their frustrations with the recent Apple updates of iOS10.1 and macOS Sierra.

MATTHEW HEUSSER: Thank you, Michael. Welcome to the show, everybody. So conference season is winding down and we’re actually doing a “regular show” for once, and not an interview. I think we should start with “the news”, as always. So my number one news segment, which we were talking about before the show started is “What is up with Apple?” I’m a Mac user, have been since… 2008, maybe, and I no longer look forward to updates. When things get updated, stuff breaks. The user interface becomes less intuitive. Things take longer. There are extra screens coming on board. I get it, Apple is trying to integrate iTunes and Apple login with all of your devices in iCloud. But it’s painful. I don’t want an iCloud subscription. My phone pops up “do you want… your iCloud is out of space”. Like, I don’t care. So? And Michael has a very similar experience, I figured I’d let him rant for a little bit.

MICHAEL LARSEN: Oh, boy… so there’s two parts to this. The first part, of course, is iOS 10. Now, to be frank, a lot of the stuff on iOS 10 is kind of cool, but I feel iOS 10 is getting very bloated. There’s a whole lot of stuff that’s coming in that I don’t necessarily need. The home screen changes… like so many things, of course, when it comes to doing any type of an update, we complain about it at first, and within a couple of weeks we forget that there’s even a problem. I’m more frustrated with the updates to Sierra, to tell you the truth. I’m also an Apple user by occupational necessity, because my company uses mac as our development platform, which is most of the time great. But I’m really finding it frustrating, especially just setting up the podcast this morning. I think I had to tell Siri five times “no, I don’t want to set up Siri. I don’t want you to help me with anything. I don’t want, every time I turn the microphone on and try to say something for a podcast, to have Siri pop up. I had to go through five or six different screens just to get Siri to shut up, and finally just disable it.

Those first couple of days, really do throw you for a bit of a loop, and unless you are just always in that super excited “ooh, bleeding edge, let’s see where we go here”, and are really good at unlearning your regular habits, yeah you’re going to be in for some frustration. I’ve got a household of five, including myself, and we all have iPhones, and everybody’s doing the update, and I’m getting some variation… well, except for my son, (he doesn’t want to admit to having to ask me anything) getting some variation, but everybody else is going “why, wait, how, what’s going on and how come I can’t get to pictures and what’s the, where’s this, and fill in the blank. I’ll let somebody else grouse [laughter].

MATTHEW HEUSSER: There’s a couple of very specific things that are going on that I think are worth mentioning. The newest update to iOS, iOS 10.10, has an extra screen, that, when you click Home, you don’t get your home screen, you get this News screen… and I think it is supposed to be faster, because it’s, like, recent apps, so the last ten apps you used are there, but I’m not used to it, and I always find that I want “must be number eleven”, so I add an extra click to every activity. I’m getting iTunes popping up in Mac Sierra and I do not know why. It just pops up. Occasionally it will ask me for the password to my iTunes account. I don’t know what that’s for. What is happening? Why is this happening? And Michael’s had the same experience; he’s also having Siri pop up on his Mac, which I’m not experiencing yet. I think it’s because I turned if off.

MICHAEL LARSEN: Yeah I actually did, I disabled Siri… and even after disabling Siri, in the course of recording this podcast and getting it set up, I’ve been asked three times “Hey, would you like to enable Siri?” Probably because I’ve plugged in the microphone and I’m speaking into it, and it’s just automatically assuming “Oh, hey, you’ve plugged in the microphone, you MUST want to use Siri” No, no I don’t!

MATTHEW HEUSSER: [Laughter] well, you know Microsoft fifteen years ago… “it looks like you’re writing a letter, do you want to write a business letter?” And it was Clippy. Remember Clippy? There’s an article by Joel Spolsky I can probably find, that when you run out of feature ideas, you get this “let’s make it more helpful for the user” feature, and I have to wonder “what happened in test at Apple?” I mean, are they just writing automated scripts, and no human is looking at it? Did they find it and point it out, and someone pointed out that “yes, that was according to the spec, yes, there’s supposed to be a new screen. Yes, iTunes should pop up when you are doing… I don’t know what. So it was in spec, but the user experience is terrible. So somebody else help me out, how does this happen?

YARON KOTTLER: Before I give my opinion on “how does that happen?”, I can definitely say that my problems have been even more basic. For me, getting into my phone has become a struggle. That whole fingerprint thing and no longer having the ability to put in the code, because that used to be on the left. That just disappeared and, sometimes, my fingers are a bit wet or something and I just can’t get into my phone. Drives me crazy. Also found interesting you reference to the Microsoft story. I think it’s both from the perspective of trying to make the product more usable, more useful like you said, but also just the complexity, the ever growing complexity through integrations through the whole other set of tools that integrate and collaborate and all of that, generates a huge compatibility testing overhead. Even if it’s within the walls of Apple, I still think that it’s just become so big. That’s just a pure logistical challenge, especially combined with the amount of time to market considerations and so on. I think it’s kind of funny that only people who were kind of trying to get stuff for free used to jailbreak their operating systems. These days, jailbreaking is almost like just getting the phone to do what you actually want it to do.

MATTHEW HEUSSER: Yeah, that’s crazy. I agree on the “too complex, too many components, too many different phones”. They’ve supported too man different versions of too many different devices that are coming out too often. When Steve Jobs took over Apple, the story is they had a product catalog that was 122 pages thick, and he couldn’t figure out what he would buy if he were a customer. So he drew a box, and he had “Business/Personal” and he had “Desktop/Laptop”. He said “we have four products. Four!” Today, if I know I want an iPad, I’m going to have a row of them, and those are just Apple products. Somebody’s got to be testing all those things, and they’re trying to increasingly get one… iOS and MacOS are trying to come together, but we’ve seen this before. That’s how we got Windows 8. It just amazes me; it’s disappointing.

PERZE ABABA: Yeah, you know my [sigh]… I guess my concerns with the latest upgrade with iOS 10 really was more around the basic stuff, too. I use the Photos app a lot, as well as the Camera app. They moved some stuff around especially when you want to go straight into editing quickly, before I post something, they kind of changed the icon, so that was… it took me maybe a minute or so to figure out “wait, what happened to my editing feature? Where did that go?” the messaging app for me, it’s… maybe it’s personal preference. I have the tendency to flip my phone to landscape when I’m messaging so I have better access to larger keys, but that completely switches to handwriting mode, which is like, switch back a couple presses back to the keyboard that I need. But, you know, as with any Apple changes, it took me a couple of days before I got used to it, but the keyboard thing in Messages, I still get annoyed with that. That’s the one thing. From the ones that I’m really hearing a lot, there’s a lot of usability constructs that we’re used to, but then, when they change that, it kind of, our brain tries to create separate neural pathways for us to get used to, you know that new flow, and then we start missing the old flow, and then we start liking the new flow, so it’s interesting how the human factor, and then how we kind of interact with the software in front of us, is a factor of how these updates are nowadays.

MATTHEW HEUSSER: So, on the Mac now, when I press and hold down the ‘a’ key, because I want a bunch of data to do input as a test, it doesn’t actually give me a bunch of ‘a’s’. Instead, I see International characters that are similar to the ‘a’ that I can select from. This is a new feature to support Internationalization. How do I turn that off? Like [laughter], I don’t know.

YARON KOTTLER: I’m sure there’s a way, however, I think what’s probably most telling is, what really gets me is the good old quality problems, which tells me probably that usability was an afterthought as well. If it was just usability and, like you say, after a few days, we all kind of stop complaining… yeah, that would probably mean “OK< fine, they’re smarter than all of us, we just need to learn”. But, like you say, why does iTunes open up once in a while when I never ask it to? Why do things happen that make absolutely no sense? Not to mention, every single person always complaining after an upgrade on “oh, the battery’s not really good”, the sort of things… unless these are bugs, I really hope these are not features, which would get me really angry, the fact that they’re bugs just tells me that, like you say, something is up with their testing and perhaps usability issues are not just them being smarter, but just them not caring.

JUSTIN ROHRMAN: I think this is actually an interesting case that supports the idea of releasing much smaller pieces of software much more frequently as a risk reduction strategy. It seems like they made lots and lots of changes over probably months to a year’s time, and then pushed them all to production. Of course, you have no clue what’s going to happen when you’re pushing that much software. There’s no way to model the software interactions, let alone the software interactions plus the human interactions, so you get all kinds of goofy stuff that happens when you get into production. Whereas, if you make a change that’s, like, two lines of code, and then actually test it, and then push that, it’s a little less likely that you’re going to have Siri asking you if you want help writing a letter every time you turn your phone on.

MATTHEW HEUSSER: That’s really interesting, Justin. There’s a couple of context things there, it’s an operating system, not a web app, so deploying is harder. You could come up with a nightly deploy or something.

JUSTIN ROHRMAN: I mean, Windows manages to ask me if I want updates, like, once a week.

MATTHEW HEUSSER: That’s true. We’ re gonna’ put an article in the show notes which kind of inspired this. The guy was saying that the deploy schedule for the integrated hardware/software for the mac products is too fast. He was actually suggesting that Apple slow down to a release cycle of every two years for operating systems and products, which would create more work in progress, more change, that needs to be tested. There’s a bigger batch size.

JUSTIN ROHRMAN: Yeah, you get the waterfall problem. I think they need to go much faster and much smaller.

MATTHEW HEUSSER: Nice!

YARON KOTTLER: What would all the marketing people do? [Laughter]

MICHAEL LARSEN: Marketing would then look to sign up people to be part of their bleeding edge A/B team. I’ll be honest, I’d sign up for that.

MATTHEW HEUSSER: You could charge for it. That’s the way they get the software as a service at the operating system model, and, as a marketer, I’d love it, because I’d get a new feature a week to sell. The home page could have a new feature a week. “Oh, you’re not on our special programmy thing? Eh, it’s going to come out in six months for you. But it’s out now!” We’re over simplifying; that’s a really hard change. Apple’s a big ship. It’s going to move slow, but there’s some opportunity there, I think.

JUSTIN ROHRMAN: Yeah, I don’t mean to dismiss the technical difficulty of doing that, but while people are dismissing it, there will be people out there figuring out how to actually make that happen.

MATTHEW HEUSSER: Absolutely… but we’ve got to move on, though. So, last week, we had in New York City Test masters Academy. Justin went. Who else is a friend of the show wo was there, Justin?

JUSTIN ROHRMAN: Anna Royzman was there, Ale Moreira was there.

YARON KOTTLER: Bernie Berger was there, I believe.

MATTHEW HEUSSER: And of course Qualitest sponsored the show. Well, they sponsor this show, but they also sponsored TestMasters Academy. Excelon Development, we sponsored, too. And the big… is controversy the right word? Conversation that it flowing out of that is “what makes a tester a tester?” There’s a couple of different angles I want to hit on that. So, when I interview people, I take them to a web page and say “How would you test it?” Maybe more than half of the SDET’s and automators say “I don’t know, give me a test script and I’ll write you some Selenium. I don’t actually come up with the test ideas. They’re already all written down, I just institutionalize them in code. I need your test requirements.” And I find that very disappointing. I’m reluctant to call that person… I mean, really simple web page. One field. One logical transformation. One button. They’re going “uhmmm, I guess I could do SQL injection on it”. What? Like [laughter] Fahrenheit to Celsius, would you try 32 to 0? So that’s one aspect of what makes a tester a… I wasn’t there, but I understand that one of the keynotes said some things, and James Bach, who was the other keynoter, replied with something like “You just said you’re not a tester” because the words used don’t match his definition. So we want to cover what makes a tester a tester but first I want to hear the story, what happened there?

JUSTIN ROHRMAN: I was actually there. What happened was Maaret was talking about her role in facilitating testing, I believe. What came into question was “are you still a tester if you’re doing that?” If you are not explicitly in the tester role all the time, are you still a tester? That’s a very ambiguous way of describing what happened.

MATTHEW HEUSSER: So she was facilitator, designer kind of person, not so much doing it?

JUSTIN ROHRMAN: Well, yeah, she was doing testing, but she was also doing things like facilitating, pair programming, pair testing and a lot of mobbing.

MATTHEW HEUSSER: Sure, she was spreading out good practices.

JUSTIN ROHRMAN: Right.

MATTHEW HEUSSER: For those not familiar, the simplest way to think of mobbing is one person at a keyboard, the whole team in the room, everyone focused on solving the problem.

JUSTIN ROHRMAN: I think there’s actually a cadence to it, too; everybody gets x number of minutes at the keyboard.

MATTHEW HEUSSER: There are rules, there’s a rotation strategy, there’s a lot more to it than that, but the basic idea is that everyone is focused on solving the problem. The idea is, testers learn about what devs do, devs learn about what testers do, graphic designers learn about what everyone does. That’s the basic idea behind mobbing, and as an exercise for an hour on Friday at 8:00 a.m. I think it’s fantastic. I’m less interested in “he said, she said”, and more into “what makes you a tester?”

JUSTIN ROHRMAN: That’s a really hard question to answer. One way to approach the question is to ask “what is not a tester?”

MATTHEW HEUSSER: About ten years ago, I was already speaking at STAR East, STAR West, Better Software. My business card said “programmer”. My role was programmer and testing steward for their department. We had no testers. The testing steward was something I made up. They had these different steward roles for, like, this piece of the database, or for that piece of the EDI. Occasionally, I’d be asked to help test a thorny piece of software. Talking to Shawn McMillen, and I said “that’s the problem with you developers, you…” whatever it was I said. He looked at me and said “Matt, do you self-identify as a tester? Do you think of yourself as a tester more than a programmer?” And that was the moment I was like “Yeah, I do.” Even though my business card said programmer. I think that I had a set of attributes and skills and attitudes that made me a tester, at that point in time. So, what makes you not a tester? I’m not sure it’s what’s on your business card. Jon Bach, who’s at eBay, Director of Live Site Quality and now Quality Evangelist. I think James would agree that he’s a tester. His day job isn’t testing.

JUSTIN ROHRMAN: I think he performs some of the tasks a tester does.

MICHAEL LARSEN: Maybe this might help add a little bit. I’ll put this in the show notes, too, so that people can read it more extensively. This is a recent article that was published on TechTarget. It is actually called “Five Qualities of a Good Software Tester. Hint: it’s not Tech Skills”. Written by Jennifer Lent. I’m not going to go through each and every one of them, but I do like some of the things she calls out here. A creative mind, Intellectual curiosity, Self-confidence. Persevere, and have diplomacy skills, and I’ll let you read the rest of the article to have that fleshed out. That’s a really good starting point.

JUSTIN ROHRMAN: I like to draw a distinction because I’m interested in strength sports. There are people who will go to the gym, and say they are doing power lifting or power lifts or whatever you want to call it, but you’re not actually a powerlifter unless you go and compete, and I think it’s similar with testing.

YARON KOTTLER: So I have a couple of thoughts about that. A couple of years ago we went through a process of codifying what a QualiTester is, and we ultimately came up with four topics. I think three of them are generic and probably true for any sort of definition of what it takes to be a good tester. The fourth one is a bit more QualiTest specific, but the first three we defined as “a tester mindset”, and that, I think, echoes a lot of the points that you raised earlier. Being a critical thinker, inquisitive, or having a lot of inquisitiveness, a strong ability to learn. In the case of QualiTest, I would say a “career tester”, I like people that have made that choice, and are comfortable with that title (not everybody is), but I think it makes a big difference when you are. Area 1 is testing mindset. Area 2 is testing skills, and that’s probably the area most people tend to gravitate. Test design skills, some tools, and so on, and I think the last area which also was echoed earlier, somebody called it “political skills” or something like that; we call it “communication skills”. So ultimately between the tester mindset, the testing skills and all the subset characteristics, I think that’s what we call a QualiTester, and we’ve really invested a lot in training, onboarding, learning courses to develop each one of these areas, and kind of build a career path.

MATTHEW HEUSSER: I think there’s a value in saying “I predominantly study test as a profession”… like when someone says “I’m a lawyer” or “I’m a doctor”, and there is a level of expertise that corresponds to that. Yeah, you can hire people cheaper than Excelon or QualiTest. You can get people that will say the right things, that are cheaper than Michael, Perze or Justin, but what’s the quality of the work that they’re going to do?

JUSTIN ROHRMAN: I think that’d a significant part of James’ point, actually, that there’s a lot of craftsmanship fostered under the role. If the role disappears, who’s responsibility is it to study and get good at this thing?

YARON KOTTLER: I think that having people that are aligned and share common values, come from an inquisitive perspective and so on, I think that alignment on values, perspective and, of course, the skills, but again, having certain people that get excited about certain things and don’t run away from a certain definition, doesn’t mean that they are only in that box, but are definitely comfortable being in that box when they can or should be, I think it does make a big difference. I see that from the perspective of how engaged people are. I see that from the perspective of how effective and ultimately how much value, how much of the extra mile people go, I also see it from the perspective of how long people stick around.

MATTHEW HEUSSER: Yeah I can see that. I think there’s value to the professional declaration. When you interview someone, to people who’ve both been testers for… one guy’s been five the other ten (years), and the person who’s been a tester for five has listened to this podcast, is on Twitter, is… just being awake and alert and paying attention, and using your downtime well and having any interest. There’s value there.

MICHAEL LARSEN: I could also throw some thoughts in here, too. One of the things that has been interesting for me. I was working as a software tester for a good fifteen years, and I was for all practical purposes, I considered myself a good tester. Nothing special; I was happy to do the work that I did, I felt like I was good at what I did, but yeah, I was on that treadmill of “how do you sell yourself? How do you tick off the boxes? How do you impress somebody with your credentials?” And I discovered, in 2010, that the best way for me to do that was to become a loud mouth, and I’m using “loud mouth” in a broad sense here. I started a blog. I got involved with AST so that I could take and then teach their BBST courses. I got involved with Weekend Testing, got involved in this little organization called Miagi-do, there’s a bunch of stuff that I participated in and, when you have a developer for example. How does a developer show that they’re a good developer? Well, they have a GitHub repository or some other method of showing their work. As testers, especially if you’re involved in Weekend Testing, it’s a practice place where you can have your name listed and you can demonstrate in real time what you can do, and it’s searchable. People can look at it and go “Oh, hey, this person shows up at a number of these events, and I’ve read a couple of their chat transcripts and, wow, they find some pretty cool stuff, they make some great comments. I might want to know a bit more about this person. That’s a little more than saying “Do you have five years of X?” It’s showing that you actually can demonstrate your skill, which I think is a cool aspect to that. Going to conferences, speaking. Every once in a while, it’s really nice ‘cause I’ll look in my Twitter notifications feed and somebody will link to a talk that I gave, maybe last year or a couple of years ago, and that percolates up and generates interest, and some of them I’ve even forgotten about, it’s been awhile and “oh yeah, that was a cool thing, and I’ve changed my thinking about that. Maybe I should do something with that, or update that on my blog, or talk a little more about that.

YARON KOTTLER: I would say that everything you just said fits perfectly with it because it demonstrates a lot of the things that are very easy to claim. You know, taking an interview, for some people, is easier than others, or going through the interview process, and frankly, having interviewed a lot of sales guys recently I can tell you that a lot of people know to say all the right things, but can’t necessarily do what they say. So I think that, for a hiring manager or for a procuring manager, for that matter, for a complete service, it’s more than just checking the boxes, it’s kind of like getting comfortable with what you can do and your track record and your reputation, and your personality, too. I think a job description should be three bullets, not a page and a half. What can we ultimately bring it down to, synthesize it down to three points, and besides that, get a bit of understanding of a person’s mindset, personality and all of that. Ultimately, in technology, it’s all about your ability to learn and communicate in any case, so all the rest is “nice to have”, I guess.

PERZE ABABA: One of the things I’ve recently realized is that, when we perform interviews on just figuring out if this tester, or a tester, can actually help us in our current organization, is that I need to kind of step back and realize “What are our developers actually doing?”, especially with the Agile or the Dev-Ops mindset where the primary focus is no longer “building things right the first time.” We’re building things with change in mind. When I’m looking for testers now, it’s looking at “how can they adapt accordingly on that specific mindset?” there’s some core values that you can look at, and primarily base things from. Yaron mentioned earlier critical thinking is definitely key. The ability to experiment, to be able to employ conjectures. I really like what Matt said earlier about “put them in a certain situation that forces them to ask questions about the product before they actually try to write test cases, or jump in and test. From a testing role perspective, I know that’s something we need to look deeper into. I know Kate Falanga has been writing about this, and I’ve been thinking some more about this. The more I look into it, I do realize that a lot of these roles that we’re looking into, they’re pretty situational. It really depends on what the need is for a given sprint, what does the tester need to do? It’s not the same thing that that person is doing over and over again. We’re not encountering the same problems that we solved the previous sprint as the things we’re trying to do again this sprint or even the next sprint. I guess the ideas of being a specialist or generalist comes to mind. Two other folks in this podcast right now went to the Reinventing Testers conference, and I know James talked a little bit about the importance of identifying “where can we generalize with and what can we specialize with?” I think that’s actually pretty key in finding out, as a person, who is hiring testers, or is in charge of interviewing people who can then work in the team as testers. I think those are key as well.

MATTHEW HEUSSER: It’s a tough subject. I think we’re going to have to come back to it. Yaron’s right, I get the “must have five years of Java” for people I don’t have a relationship with yet. They don’t know me. They’ve got it all figured out. They’re just coming to me to fill a req. That’s not really what we do. I think we have room to explore that… but, we have to move on. Now the mail bag for the testing show is… what’s the email address?

MICHAEL LARSEN: The Testing Show at Qualitestgroup dot com, and with that, it’s time to talk about upcoming events. Who’s doing what, where, when and how?

PERZE ABABA: By November 10 and 11, I will be in Philadelphia for Test Bash Philly. MY company is actually sponsoring this. The Ministry of Testing folks just announced this, so if you want to say “hi” to me and talk to me about the show or testing in general, please reach out.

YARON KOTTLER: Yeah, I’m going to… it’s not a testing focused event, it’s an industry event it’s called RSNA. It’s the Radiology gathering. One of the industry groups we’re also involved in is DICOM, which is the standard committee for document imaging, and basically all of radiology. We definitely attend every year, we’re part of the DICOM standard committee and that’s kind of our opportunity to participate in what’s going on in that industry.

MICHAEL LARSEN: I’m not gonna’ be appearing anyplace new, I’m actually winding down the year kind of quietly; this year was an eventful one for other reasons, and I’ve mentioned it in the show in the past. I do want to give some shout outs that I think are relevant to this, and also I want to give a salute to a mentor of mine, so if you’ll bear with me a bit… Carina [C.] Zona did a keynote at PyCon in Australia called “Consequences of an Insightful Algorithm”. This is something that I jumped on and I found really interesting, I definitely recommend checking this out because it’s all about –what you might want to think about for being a tester- what happens when machine learning, what happens when are algorithms, for the best of intentions, go horribly wrong and do things that are just… everywhere from mildly amusing to grossly inappropriate? It was a really cool keynote and there have been a couple of podcasts; CodeNewbie has done a session on it. Ruby Rogues has done a session on it. It’s a really neat topic and I encourage people to check it out. And the final one that I want to give, it’s kind of a send-off for me personally. I’ve lost a mentor in my own company, but for probably the coolest of reasons (and Matt will appreciate this). This month we said goodbye to Audrey Tang as a principal developer for Socialtext, but I’m not shedding too many tears because she has become a Minister without Portfolio in the Taiwan Executive Yuan. I don’t know how you compete with that but that’s some kind of exciting news and Audrey has been probably one of the best mentors to me the past four years that I’ve been at Socialtext… and if you don’t know who Audrey Tang is, I’ll set up a link for the stuff that she’s involved in. She’s kind of amazing! If you work in the Perl community, at all, you already know who she is. If you don’t, you still should look her up. It’s just incredible what she’s been able to do and how much I’ve been able to learn from her.

MATTHEW HEUSSER: Yeah that’s, that amazing. I worked with Audrey when I was at Socialtext for a couple of years, too. As for me, I’m going to be in Nuremberg, Germany first week of November for QS-Tag and Pottsdam, Germany the second week of December for Agile Testing Days. And then Agile Testing Days next June, but nothing else really on my plate coming up. I’m trying to be home a little bit more. There’s a couple of events in Ohio January and February time frame. I haven’t decided if I’m going to go, I haven’t put in for a talk, I would just show up and be Matt… so if anyone wants to see me somewhere, send me an email, but that’s really all we have for now. Thanks, for listening, and we’ll see you in two more weeks on the next episode of the Testing Show.