Making QA Strategic

April 28, 2016

This week, Matt takes The Testing Show on the road, or more specifically, the Highway. At QA or the Highway held in Columbus, OH, in March 2016, Matt was a panelist for a discussion on Making QA Strategic.

This is a live recording from that event. From testing and its place within an organization, How does testing fit into an overall IT strategy?

Transcript:

MICHAEL LARSEN: Welcome to The Testing Show. I’m Michael Larsen, the show producer. This week, Matt takes The Testing Show on the road, or more specifically, the Highway. In March, QA or the Highway held their annual conference in Columbus, Ohio, and Matt went as a speaker and as a panelist for a roundtable discussion on the possibilities around Making QA Strategic. Today’s podcast is an excerpt of that roundtable discussion, recorded live and in real time, so without any further ado, let’s join Matt and the rest of the QA or the Highway Panel already in progress…HOST: This session is really to focus on testing and its place and the strategic visioning within an organization, and so we really want to focus around that, “How does testing fit into an overall IT strategy?” We call this the Great Lakes Panel of Testing Leaders up here. We’ve got folks from Canada, from Cleveland, folks from here in Columbus, Michigan. So, we can take 30 seconds in brief introduction of who you are, your role in your organization.

JOSH ASSAD: I’m currently a Dev Manager at a company called Desire2Learn in Kitchener, Canada. Before that, I was a tester for many, many years.

KEVIN MALLEY: I’m Kevin Malley, a QA Manager, at Economical Insurance in Kitchener, Ontario, Canada. I recently started there. I’ve only been there a few months. Before that, I’ve worked at a number of other high‑tech companies in the Kitchener, Waterloo area, mostly around software testing, but project management, Agile coaching, and etcetera.

JARED SMALL: I’m Jared Small. I’m the last Canadian on the panel here. I work at BlackBerry as a Test Manager.

MATTHEW HEUSSER: I’m Matt Heusser. I’m the Managing Director at Excelon Development, LLC. We’re a small consulting company in West Michigan and everywhere. Consulting and training on software delivery. We do contracting and a lot of writing.

DIANA WENDRUFF: Hi. I’m Diana Wendruff. I’m a Manual Tester at a local consulting firm here in Columbus. We do Adobe Experience Manager implementations for some of the big companies around the area.

ERIK DAVIS: My name is Erik Davis. I’m the Manager of Testing at Hyland Software in Westlake, Ohio. I’ve been there for, actually, 13 years and 13 days. My job, in short, means I own our internal QA training team, the piece of automation that sits within test, as well as trying to looking at our entire testing process.

HOST: I think this is a very, very important conversation for all of us to have and understand, and let’s just get right to the meat of it. You know, “Does testing belong at the strategic table? Should they have a seat at the table when it comes to IT strategy along with a PMO group, an architecture group, that we traditionally see making these big strategic IT investments? Should testing have a seat at that particular table?”

KEVIN MALLEY: So, I think, absolutely. We all have a vested interested in the products that we build, and we also have vested interest in the people that we have in our departments. I personally focus a lot of the people, try and get them involved, and I think that’s a good place for those people to get involved. I think they have a lot of value. I know, in my past experience, when I’ve brought people into more strategic conversations, they’ve shown a lot of value. They ask questions that I don’t see being asked by others lots of times.

HOST: So, a follow up to that, I think the natural question is, “What is that value that we provide? If we should have a seat at the table, what is that strategic value? How do we actually influence strategy and provide that?”

MATTHEW HEUSSER: So, two pieces to that. If you are in a traditional organization that doesn’t ship software all that often, over time, the amount of unknowns increase between your ship dates. Its unknown space is, “I don’t know.” And then, you ship and then your unknowns vastly decrease because it’s on production again. So, in a traditional environment, who’s going to answer the question, “What’s really going on?” Testing can do that. That’s kind of an operational trusted advisor. That’s the voice that can speak to the vice president of software engineering. But, if you’re getting into the more modern techniques, you know because you’re shipping every two weeks, and you can project based on data. So, this value that we used to hold onto that we could tell the truth, is much less valuable. A couple of things testing can do: We can advise on the process, the capabilities of the system, and the stability of the system. For example, I worked with a company. I had about six teams, and one of the teams had to do the whole mandatory-overtime, work-really-hard, create-the-deadline thing. And, that team was essential. They never shipped anything. Over 50 percent turnover during the project. Over 100 percent turnover within 1 year of the project. That team basically got nothing done for the next two years, because of the strange institutional behaviors. The stability of the company was undermined, and all of us in test new it. You’d overhead it at the water cooler. So, the question is, “What is constraining this information to getting from senior management, and why don’t they understand it?” And, if we could’ve gotten that information delivered in the right way with truth to the right people, I think that would’ve escalated QA to a strategic value, which we’ve lost by going to something like Scrum because it just was much more transparent.

HOST: So, to piggyback on that—right?—because I think what a lot of us probably heard in that statement is, by going through Scrum and some other, you know, Agile development lifecycles, because what we release is more known, what you’re saying is, some of our value that we used to provide is gone? Is that what you’re saying?

MATTHEW HEUSSER: The traditional value is less relevant. So, if we want to stay relevant, what we do has to adapt. So, one of the ways we could do that is by commenting on the stability of the overall system—whether that’s the deliver process or the people or whatever else it is—actually assuring quality.

HOST: So, the piggyback for the rest of you out there, I mean, how is testing viewed in your organization, and how has that changed over time?

ERIK DAVIS: We have a very high – we have a lot of testers. We’ve been for the last 13 years—well, forever—been approaching a 1:1 ratio between test/dev because our CTO, which is like he’s at the right level because he pays half the company’s checks, decided that he wants lots of testers, which is helpful. So, we don’t have fight for our existence, but we’ve had to fight to show that we can provide, I think, strategic value. So, we were not exactly a step‑at‑the‑end and stuff got chunked over the wall, I guess, because I’ve worked places where it was like that and you had no communication. We still communicated with devs all the time, but it was more of a, “Once dev got done, then whatever time was left between dev-done and ship was how much time you had to test, before we shipped, and then you did somewhere testing after you shipped because there’s still problems and we’d like to find it and fix before lots of people buy it.” So, I think we were viewed as an important piece; and, even though it was stated that, “We aren’t responsible for quality,” we were used as sort of the gatekeepers for a long time and that’s come and gone a few times, I think, over history, which is good. Because, I don’t think that test should be the ones to decide that, “Yes. This ships.” Or, “No. It doesn’t.” And, we’re heading back in a direction of being able to provide information to dev so that the people who are making the thing and are involved in, like, the people who are paying us to get it can make a decision based on what we’re providing them on whether, like, the current state is good enough to ship or not.

HOST: So, to kind of piggyback, if our traditional role has changed and where we used to provide value has changed, because Agile methodologies have caused us to be more collaborative and more transparent. “We have role sharing, right? People say, when we have teams, you know, “Testing is everybody’s responsibility,” sort of thing. What are some strategies that we can employ? How can we as testers demonstrate value to an organization so that we can continue to be in existence, or is the Yahoo way the way to come? Should we all be searching for new careers? Who would like to take that one on? I’m not paying for days out here.

JOSH ASSAD: So, I think that the testing organization has, testers in general, have responsibility to change up, just like anything. Along with technology in general, you have developers using new tools that are expected to be up-to-date on the latest technologies. So, why shouldn’t we as testers as well? So, certainly, testers need to evolve beyond testing features, beyond testing, you know, domains. They need to develop a system-wide testing skill, systems thinking, and I think it’s that kind of thing that gets lost sometimes for even at a product level where product managers are interested in implementing a feature, developers are implementing things. In an Agile methodology, things are moving quickly. So, you don’t have, you know, “How does that new feature impact this other feature?” You know, “How did the systems, when the systems come together?” That’s where we can offer a lot more value as testers and test managers and people making strategic decisions in organization. If you have a team, in my opinion, that’s where you should be grooming your team to be. I think Karen also mentioned it as well today, “Getting involved in those discussions with your product management team, with your architectural team.” Not just being in the room, but go in prepared and be thinking about how this feature or this way of implementing things is going to impact the system as a whole.

MATTHEW HEUSSER: Fantastic book, Gerald M. Weinberg, Quality Software Management, and Volume I, is called, Systems Thinking. It’s a little dated. But, it’s so good, you should read a book that’s 20 years old. Have you ever been in a meeting where someone says something like, “We should count the number of change control requests and score the team for the most change controls,” and you say, “If you do that, the change controls will go up. I don’t know if productivity is going to go up,” right? So, there are unintended consequences, and a lot of times people can’t see them outside of their team or their workflow or their process, “Oh. In a mature organization, that’ll never happen.” Even people giving presentations have said that—managers and directors. I talked to my network and talked to someone at the company, and they laugh and say, “Yeah. That happens.” So, I would say, “Systems Thinking,” is the ability to model the key behaviors of the system so that you can predict the consequences, so you can choose the consequences you like the best. People have degrees in this. There’s academic subjects in this. There’s much better ways to word it. But, if you want the super-high level, when I say, “systems thinking,” that’s what I mean.

JOSH ASSAD: I just want to add one thing to that. I think it’s more than just the modeling. I think it’s a deeper understanding of what it actually is and how it works. That’s how I view it a little bit more than just seeing a pictorial.

HOST: So, we almost identify it as, “Systems thinking is probably a skill that we need to develop to provide ongoing strategic value to the organization.” What are some other skills that we either need to continue employing or we need to develop in order to continue to provide strategic value?

JOSH ASSAD: I think the other thing that we need to do is build partnerships. So, some of the comments that we made earlier, a couple of panelists made earlier, was how we’ve evolved over time. What I’ve seen is that previously we didn’t have to react as quickly. So, if we got ourselves in early, at the table, we would discuss “something” and some months later, we’d actually do something about that “something” that we talked about months earlier. Now, it’s more of a just-in-time model, and it’s really important—or at least I’ve found it’s really important—to build partnerships with those people that you work with so that you always have somebody to go to. The network of people is always what, you know, helps you out. We’re never going to be an expert in everything. So, that network is best. I try to build partnerships with my customers—whether that’s an external person from the company or an internal person, it might be another department—build partnerships with them, to help them understand what we are about, what value we can provide, and also the partnership goes the other way so that they can then teach me what they’re looking for and what they’re about.

HOST: There’s kind of an unwritten thing within the testing community that we all kind of acknowledge. We don’t actually create software. We’re not sitting down, we’re not writing the codes, and we’re not giving birth to this thing that becomes an application. We’re the people coming in and effectively criticizing, “Not working right. Not working the way we think.” We have an inherently critical job, but our job isn’t to create. So, when you talk about providing value, if we’re not creating something and we’re always criticizing, what do we need to do in order to provide value to the organization?

KEVIN MALLEY: We’ve had a lot of great on our team when one of our testers basically started paying more attention to change logs and the build system. Basically, it showed us that we had another oracle that we could rely on that was a lot more accurate and more honest than our work tickets. There were a lot of changes going in that were introducing problems that we were completely blind to before we started looking at the commit logs. Even if you don’t want to recode or learn how to code, understanding how the system gets put together and where these other sources of information are is very important.

HOST: One of the things that I’ve noticed within the testing industry is that in many ways it mirrors what happened in the steel industry. Initially, all the manufacturing took place in the States. All the mining, converting of the ore to steel, that happened in the States, and then it became very, very cheap to do that overseas and the entire steel industry really didn’t pay attention to the shifting marketplace. And, they really didn’t adapt, you know, the steel industry is all but gone in the United States. And so, we’ve all seen it with the outsourcing of testing and testing services. Is that indicative of value that we are failing to capture by outsourcing it, or is our job as such that anybody can do it anywhere?

JOSH ASSAD: I think it’s primarily a business decision. Hopefully, they have our input when they do it. But, what I can say is that we’ve seen instances in our community. Both startups and larger companies, in both cases, I’ve seen outsourcing happen. I’ve seen testing come back to the main company. In my opinion, it’s not really a concern. I think we have to show our value all the time. It’s just built into our role, and we can expect to see these kinds of peaks and valleys in outsourcing. I think when it comes down to it, when a company is releasing something critical, my experience is, a lot of times they’ll bring the important back in-house.

MATTHEW HEUSSER: So, a couple of things: Outsourcing for pure cost reasons, means people don’t see your value, “All we can focus on is a nickel.” So, I’m not a big fan of that. But, if you’re company says, “We’re not really good at testing. We don’t want to develop the expertise. We want to find a long-term partner that can help us with this, that hires people who really love testing,” and hopefully your partner does things like, “We’re going to look at turnover, so our contractor can develop the subject-matter expertise.” That’s a very different way of looking at it than, “We’re going to save a nickel to outsource.” When you focus on cost, values go down, and there’s some empirical research that costs actually go up because of something called, “Failure Demand.” That’s where you call the 800 number and they can’t solve your problems, so you call them back. So, they have two transactions instead of one.

The second piece I wanted to cut is that we can have insights. Subject matter expertise I think is really. I can give a story, a true story. I was at an insurance company. This was a few years ago. We had just gotten social security numbers by law, and somebody says, “Field 1 is Social Security and the Field 2 in the Member ID.” And, I said, “No. Field 1 is Social Security, unless they join after a cutover. In which case, Field 1 becomes Member ID, and Field 2 becomes Social Security.” “Yes, it is.” “No, it isn’t.” “Yes, it is.” “No, it isn’t.” “Yes, it is.” “Why are we arguing about a fact? Pull up the data extract. Let’s look at it.” “Gee Matt, I’m surprised. You’re right.” If we hadn’t gone through that exercise, we would’ve had a bad requirement. We would’ve made a bad assumption, which would’ve been checked wrong, which would’ve been a defect in production. If you can do that consistently, especially early in the process, testers can often be really good at it. This is what the software should do when the product owner is way up here in the vision. These 25-cents-on-the-dollar outsources, they can’t do that.

DIANA WENDRUFF: My company current company also has offices in India and whatnot and we try having team members from there on our team, but since the time zones are so different it became increasingly difficult to even communicate in leaving notes sometimes. Even though they were detailed enough, they were still very hard to tell someone how to do something. For my project now, everyone is actually in our office for most the part, and it’s a lot easier to go over to the developer and say, “Hey. There’s this.” Or, “Hey. Can you come over to my computer? I need to show you something.” And then, you know, the occasional lollipops, they help too.

HOST: I do want to piggyback onto something, because, Matt, you talked about “Failure Demand.” Obviously, at some point, somebody is making a decision about whether or not to outsource or not. So, how can we educate the powers-that-be about the value that we actually provide?

ERIK DAVIS: For us, I think it’s not too difficult because our testers do a lot of things that are outside of testing but I think are relevant and add value. So, because of our place between types of knowledge with dev and support with the product, our test teams have a better understanding of at least a larger chunk of the system. So, they get involved in support calls. If it doesn’t require reading code, they’ll call us saying that, “Support can’t handle it themselves.” We also get called in to do a lot of the technical customer training that’s beyond the level of what our Training Department can handle. We have a lot of really good people in that department. They handle a lot of the entry-level courses and some of the specific technology ones; but, when it’s something brand new, it hasn’t existed long enough yet for them to learn it. So, they come to us to say, “Hey. Can you run this class at the User Conference, and then next year you can show us what you did and then we’ll take it over?”

MATTHEW HEUSSER: So, those are additional services—

ERIK DAVIS: Yes.

MATTHEW HEUSSER: —test and quality could deliver?

ERIK DAVIS: Yeah. And, things that we do that make it easy for them to see some of the value of what we provide.

HOST: How does automation fit into an organization from a strategic perspective, given that most of the stuff that you’ve already put up on the board is more about intellect, knowledge sharing, and collaboration? How does automation fit into that? Should it be part of our strategy? And, if it should be, how do we convince others that we should do it without saying that, “We could automate ourselves out of existence?”

DIANA WENDRUFF: So, on my current project, right here is my co-tester. He actually does all the automation for our project, and I handle all the manual. And, he’s working ON more of a regression suite at getting those things done. So, but whenever I go to do regression, I can just look around, but he can make sure the code is correct, and he handles the developers more. And, they’re all in the coder view so that I can understand it more and if I have a question, because if I point something out, they go, “Oh. I didn’t think about it that way.”

JARED SMALL: As the test leader, I have an obligation to understand the technology and work that my team is working on. I want to know what the team is doing. I want my development team and the rest of the organization to know that I see value in it as well and that I’m a part of it. Additionally, recently, I had an experience where another development team who only had automation-style testers, came to my manager and asked, “Automation style testers are not finding the bugs that your group is finding. We need help.”

ERIK DAVIS: I believe the value of automation is that it frees the testers up from having to do that by-rows checking that Michael Bolton likes to talk about where you’re just really validating that, “Yes. The system does what I expect it to do.” By freeing up your testers’ time from worrying about the day-to-day, they can focus on the more creative and more interesting testing.

KEVIN MALLEY: When you start automating, you become a developer. You know, we’re talking about QA going away and being replaced by automation, and I think you guys raised some good points. Things that you don’t catch. There’s experiences the automation doesn’t catch—timing, condition, and things like that. But, as an automator, when I’m automating a test, first off, I’m a tester, because I’m figuring out, “What’s the expected behavior? How do I capture that in automation?” So, I’m being a tester there. And then, as soon as I start automating, all that’s out the window. Just another dev, with all of my biases, my code’s great, and things like that. So, there’s still needs within automation. To me, there needs to be that QA mindset, and I just have a hard time believing that people can hold both of those approaches to things in their mind at the same time. But, I do believe in automation. You still always need QA. You still need the test mentality. If you are a good tester who’s automating, you’re still just a dev at that moment that you’re doing it. So, I don’t think that automation is going to ever replace QA. I think QA is even more important as automation. Otherwise, all you have is a fast grown-in test that doesn’t really test what you want.

MATTHEW HEUSSER: I’m a big fan of something called, “The Swiss Cheese Model of Risk.” Think about a block of Swiss cheese. It has holes in it, but the holes don’t go all the way through it. So, there’s no way you could put a pencil all the way through, because it’s going to get hit by something. So, on these continuous delivery models, we’re going to have human testing—maybe not as much. We’re going to have automation—not 100 percent. We’re going to have continuous monitoring of production. And, all of these add up as risk management tools so that any problem that emerges will be hit by something. And, you do a little bit of math, achieving 100 percent coverage, is really expensive and you never get there anyway. So, it’s basically, “What are a series of techniques we can have to get 80/20 for each, which add up to more than 100 percent in any individual one. I think about it in in terms of tradeoffs and overall coverage for risk as opposed to 100 percent automation, 100 manual, and then you can start to have conversations about what coverage means and that any moment you spend writing code is a moment you’re not exploring a possibility. Then, you make it transparent to management. All of the sudden, you’re talking about “test strategy” with senior management.

KEVIN MALLEY: I’ll quickly add to that. In the last organization I was in, we built a product from the ground up and that new team. So, we had an opportunity to kind of instill quality from day 1. So, one of the things we did was, we focused on things at the development stage, more so than we did in the latter stages, which would kind of be what we would traditionally have seen assessing the past. So, we implemented things like code quality analysis. We really focused on unit testing. But, where I thought was interesting is, we morphed the unit testing into more than just the developers. We started to pair testers and developers in the testing of that piece of development work; and, as the testers got more experienced, they’d actually write that unit test. So then, that became part of the code and that was actually used time and time again in the future. And then, as we got further along, we would focus most of our automation testing on API. Again, brand new product. We had the opportunity to kind of do it right. At that point, we had absolutely no UI test whatsoever from an automation perspective. That was all done manually. Our intention, before I left, was that we were going to add UI test to help us with the continuous integration piece.

HOST: What are some of the things—some of the old habits—that we might still carry over from an old way of thinking that we continue to do that we should stop doing? What are some bad habits that we might have that we might not realize are bad habits?

DIANA WENDRUFF: I would think, as Karen said earlier, “doing everything yourself.” Whenever we went through a slow time, during the month of December, a lot of the devs helped us out with test cases, and they would come to us, myself or Brian, and ask, “What can we help with?” And, they would ask, “Well, how am I supposed to test this?” And, they would come back and they would share what they did with us. And, it was actually a lot of help, because they saw stuff a little bit differently. But, we still went and we looked at it too. But, they understood more of what we did and how we saw, so it was helpful.

MATTHEW HEUSSER: I’m going to make some terrible generalizations, but we used to really focus on finding the bugs, “We’re going to make the programmers cry. We’re going to break their stuff.”

PARTICIPANTS: [LAUGHTER].

MATTHEW HEUSSER: And, it’s kind of a frat-boy mentality and the older I get, I see less and less value in that mentality. There still are a few places where you can do it politely. You know, “Let’s just do a bug con.” That can still have value, but less and less. Or, you could say, “My job is to provide information to the stakeholders about the status of the system,” and that might just be more than product. The process. The project.

JARED SMALL: I think I’ve actually heard folks use the phrase that, “Folks should be information brokers and not just testers.”

MATTHEW HEUSSER: I’ve heard you say that.

JARED SMALL: [LAUGHTER]. So, I think if you stay focused on your customer and you keep in front of your mind the capability that your software is providing, that’s the meat of your business, and if you can advocate that to your team and keep that in the front of their mind, then that’s a value you’re providing. You know, developers will go so caught in the technical, “How do I make this thing work,” that they might forget, “Why are we even building this thing?” I think, as a tester, you’re closer to the customer and you can bring out perspectives.

JOSH ASSAD: So, I mentioned “partnership” earlier. One of the things that I did to focus on the customer piece is, I booked a flight, went down and visited the customer for a week, worked with them, and had them show me how they use the system and some of their expectations. One month later I came back with a bunch of proposals. And, one of those proposals was, “You guys do UAT testing and you use this tool. Would you mind if I use that tool to communicate with you? So, when my team finds issues, we’ll log it in the same tool.” We use the same tools to show our progress of work as well as our progress of issues and finding issues, and it just opens the doors to, “Wow. We can be more transparent.” The second thing I’ll add is, innovations. We haven’t talked about innovation here today. But, one of the things I’ve done in the last couple of companies is really looked for ways to inspire innovation and whether that’s getting some people together to chat about whatever or if it’s coming up with something a little bit more structured, I use those sessions to help guide the technology. So then, the team is responsible for the technology and how things come together and drive it forward. I find that if the team comes up with something, they have a passion for it, and they’ll help drive it.

JARED SMALL: As a leader in my organization, I find that I have a responsibility to stay up-to-date on what’s going on in the testing community. So, personally, I stay on social media trying to keep track of the latest goings-on in the testing community. Coming to conferences like this is a great thing to do. The second part of it, within your team, what we’ve done at Blackberry is model our users into personas, and I know we’re not the first people to do it. But, that’s been a real benefit to the testing team. It’s one of those techniques that you can add in to help mitigate risk.

MATTHEW HEUSSER: One of the things that we’ve been doing for the past several years is, we come into an organization and we do an, “Analysis.” It’s a gap analysis, “What are your strengths, weaknesses, opportunities, and threats,” right? And then, “What do we recommend you do about that?” So, when you get a new manager, you get a new director. You change teams. You change positions. You get a promotion. You want a promotion. You’re doing your annual goals. You can go to your management and say, “I would like to do a gap analysis.” And, if they say, “Yes,” because what you’re going to do is you’re going to deliver to your manager what they need to accomplish their goals for improvement. Everybody wins. The other is, “Look for hidden opportunities to create value,” which I think is a lot of what this is about. We have these skills we can deploy in ways which other people haven’t thought of.

DIANA WENDRUFF: I have to say, “Communication,” because you’re not communicating, you’re not collaborating, whichever means that means through—your ticketing system, face-to-face, and e-mail. There’s different ways of doing it, but some are easier than others. But then, if you’re not asking questions, you’re obviously not communicating. And, you know, if you’re asking questions or thinking new things then you can’t know. Someone else can’t think something that you might think and you might be annoyed but they don’t know you’re annoyed, and so you just always have to communicate.

ERIK DAVIS: So, expanding on earlier, we talked about aspiring innovation or being engaged with the community. So, going to conferences, paying attention to Twitter, whatever that means to you. I think a step beyond that is not being afraid to try new ideas that you find outside of the bubble that is where you work. Because, I mean, even if you work with smart people or hundreds of people, they don’t have all the really good ideas. Taking ideas from outside and trying them out, potentially modifying them. Another, I think, is, finding whatever ways you can to get other people within your organization to understand who you are and what it is you do, and that can take on all kinds of different avenues. But, getting people outside of test to understand what it sort of means to be a tester, but then also on the flipside of that understanding what it is that they’re doing that you can empathize a bit with what they’re going through.