QA

I grabbed this one for its relevance to what we’re currently working through. It builds upon the principals from Lean Startup, which I haven’t yet read. In hindsight, I probably should have read that first, but this book stands up pretty well on its own.

Who would I recommend it to? Anyone with their hand in UI/UX design, which should be everyone out there.

The premise is that designing a user experience is difficult to do in a lean environment. All too often we get caught up in the old BDUF approach. Collaboration is the key and it starts at the design phase. Someone from every discipline of your team should be involved from the start to the finish. This establishes ownership and understanding of the goal and product. There’s no more “Oh, my job is done, on to the next thing and I’ll half-assedly glance back at this other thing now and then.”

One of the key suggestions for getting great UX in a fast moving environment is to design toward outcomes rather than a backlog of must-have deliverables. Similar to how a backlog is created (or how one should be created), customers are polled. Rather than having one person responsible for that interaction, the entire team is. How? Regular user acceptance testing. Get the product in front of actual customers and see what they say. Have everyone view the results. This book and idea are primarily focused on UX, but it can go for any feature of the product; getting more people involved in understanding customers can never hurt.

Just don’t do it… From the start, create a 2nd release. Give it a real version number.

You’re not going to get it right the first time.

Don’t try to. Make the roughest of plans and iterate. Don’t be afraid to ship UX debt, but do be sure to note it in the backlog and address it appropriately.

To generate the best solutions quickly, you must engage the entire team.

And do so in a meaningful way. Don’t e-mail out a spec and ask for comments. Call out specific people for comment. The project manager should ensure that someone from each group has signed off.

Working software [is better than] comprehensive documentation.

[Respond] to change over following a plan.

Don’t code to a spec that you know isn’t perfect. Just like your code, it will never be. Roll with it.

“Lean Startup processes reduce waste by increasing the frequency of contact with real customers, therefore testing and avoiding incorrect market assumptions as early as possible.”

[Minimally viable products don’t have to be made from code]. Each design is a proposed business solution — a hypothesis. Your goal is to validate the proposed solution as efficiently as possible by using customer feedback.

Their involvement must be continuous, from day one of the project until the end of the engagement.

This is in regard to folks from different disciplines in your team.

Keep your teams small — no more than 10.

Understand what the users are doing with your products and why they are doing it.

The success or failure of your product isn’t the team’s decision — it’s the customers’.

Rockstars, gurus, ninjas, and other elite experts of their craft break down team cohesion.

The answer to most difficult questions the team will face will not be answered in a conference room. Instead, they will be answered by customers in the field.

Frequent failures lead to increased mastery of skills.

The team’s focus should be on learning which features have the biggest impact on the their customers.

Our goal is not to create a deliverable, it’s to change something in the world — to create an outcome. We start with assumptions instead of requirements. We create and test hypotheses. We measure to see whether we’ve achieved our desired outcomes.

The hypothesis statement is the starting point for a project. It states a clear vision for the work and shifts the conversation between team members and their managers from outputs (e.g., “we will create a single sign-on feature”) to outcomes (e.g., “we want to increase the number of new sign-ups to our service”).

None of your metrics will be meaningful if you don’t have a benchmark in place.

It’s much easier to pivot from a failed approach if you haven’t spent too much time laboriously documenting and detailing that approach.

Nobody knows the customer better than the support team. They should have regular views and chance to comment on what you’re working on.

Your organization needs to adopt a mantra of “competencies over roles.”

Break your big teams into what Amazon.com founder Jeff Bezos famously called “two-pizza teams” (http:// www.fastcompany.com/ 50106/ inside-mind-jeff-bezos). If the team needs more than two pizzas to make a meal, it’s too big.

As it turns out, my weak understanding was mostly wrong. The entire strategy is based upon testing “tour” metaphors that can be visualized by thinking of yourself as a person “touring” in a new city. That new city being your code. What I did understand and agree with fully in both mind and practice is the idea that strict adherence to a detailed test case or test plan is not efficient, not enjoyable, and metaphorically speaking, not the best way to see town. Instead, write test cases that are more of a general guide and do your own exploration in order to introduce variance with each run. The tour metaphor is ever-expanding and will be different for each team and each feature. The author presented his list of favorite tours, which can conveniently be found on MSDN (the author used to [maybe still does?] work at Microsoft and most of this book was MS-centric).

Behind most good ideas is a graveyard of those that weren’t good enough. This is a universal. Don’t be afraid to screw up/do something worthless because it is all part of the process.

The modern practice of manual testing is aimless, ad hoc, and repetitive.

Software is peerless in its ability to fail.

Software is not, and likely never will be, bug free.

There is no replacement for the tester attitude of “how can I break this.” Any development team that thinks they can get away without dedicated QA persons is fooling itself. I’ve seen this time and again in the gaming industry as I’m privy to many an alpha and beta from knowing folks in the industry and my previous experience running a major gaming site. I’ve not yet been close to a project that ignored QA and did not fail. Not to say they don’t occur, but my personal experience is a goose egg. It is too hard for someone who was in the code to step back and say to themselves “This sucks.” You need someone disconnected, but fully respected to say that.

The less time a bug lives, the better off we’ll all be. Bugs found in design are the best kind.

If testers are judged based on the number of tests they run, automation will win every time. If they are based on the quality of tests they run, it’s a different matter altogether.

Automation suffers from many of the same problems that other forms of developer testing suffers from: It’s run in a laboratory environment.

Test automation can find only the most egregious of failures: crashes, hangs (maybe), and exceptions. … Subtle/complex failures are missed.

Manual testing is the best choice for finding bugs related to the underlying business logic of an application.

Tester-based detection is our best hope at finding the bugs that matter.

Exploratory testing allows the full power of the human brain to be brought to bear on finding bugs and verifying functionality without preconceived restrictions. Don’t let yourself get bogged down by manual scripts. THINK. Otherwise you might as well be a robot and then be automated.

EPIC DIGRESSION: I’ve never had luck outsourcing testing to India and this is the reason why. The workers there are not encouraged to think, and instead treated and expected to function as worker bees. Follow instructions to a tee, don’t raise your hand, put in your hours, and get paid. I was involved early in outsourcing and several stories from colleagues lead me to believe it has changed for the better, but it was so bad for me that I still have that filthy taste in my mouth and would never proactively attempt to implement Asian testing for my team until I saw it work first-hand somewhere. At a recent QA Meetup, a colleague at Symantec actually mentioned they have it working really well there. It involved A LOT of effort from both sides of the ocean. I think that was what we missed in the early days. We assumed we could just drop things on these people and they’d get it done. Outsourcing with European countries now, I don’t know if it is just the individuals or the culture that matters, but we get amazing work from a group that considers themselves part of our team, company, and culture. It is a relationship that has been cultivated over many years and the fruit it bears is wonderful.

Exploratory testing is especially suited to modern web application development using agile methods. … Features often evolve quickly, so minimizing dependent artifacts (like pre-prepared test cases) is a desirable attribute. … If the test case has a good chance of becoming irrelevant, why write it in the first place?

Having formal scripts can provide a structure to frame exploration, and exploratory methods can add an element of variation to scripts that can amplify their effectiveness.

Start with formal scripts and use exploratory techniques to inject variation into them. In my case, I do this when regressing a feature. On initial testing, I stick very closely to my script. With subsequent runs, tracing those same code paths is unlikely to discover anything new. I inject this variation then and use my former test plan to guide me to all the code, but I will not take all those same steps.

We need the human mind to be present when software is tested.

Testing is infinite; we’re never really done, so we must take care to prioritize tasks and do the most important things first.

Don’t allow stubbornness to force you into testing the same paths over and over without any real hope of finding a bug or exploring new territory.

It is good to keep in mind that most developers don’t like writing error code.

No matter how you ultimately do testing, it’s simply too complex to do it completely.

As testers, we don’t often get a chance to return at a later date. Our first “visit” is likely to be our only chance to really dig in and explore our application. We can’t afford to wander around aimlessly and take the chance that we miss important functionality and major bugs.

Tours represent a mechanism to both organize a tester’s thinking about how to approach exploring an application and in organizing actual testing. A list of tours can be used as a “did you think about this” checklist.

Despite my plentiful notes, I didn’t really care for this book. I feel it could have been summarized in 40 pages or so. The touring metaphor is a great one that I’ll keep handy in my tool belt, but the book had a number of user stories (don’t think Agile) from using the tours that didn’t really add anything, there was a good amount of general testing knowledge that is always a nice reminder, but better suited for a different book, and it finished with this wildly optimistic/futuristic section on the future of testing that had nothing at all to do with the subject I cared about reading. I had a fair amount of difficulty getting through this one, but I’m glad I did because ultimately it will make me better at what I do.

Good read. Short enough that it can comfortably consumed on a flight across the country (I did it in two because Candy Crush has got my attention).

Anyway, my takeaways (personal comments or clarification in italics):

The most important thing you can do is to just understand the basic principle of eliminating question marks.

For most of us, it doesn’t matter to us whether we understand how things work, as long as we can use them.

Many designers tend to underestimate just how much value conventions provide.

Innovate when you know you have a better idea (and everyone you show it to says “Wow!”), but take advantage of conventions when you don’t.

Making the choices mindless is one of the main things that make a site easy to use.

The main thing you need to know about instructions is that no one is going to read them.

The concept of Home pages is so important. Every site needs a landing page of sorts that makes sense of the whole.

It’s no fun feeling lost. Navigation that is clear and logical is very important.

If the navigation is doing its job, it tells you implicitly where to begin and what your options are.

Every page needs a name.

The name needs to match what I clicked.

The most common failing of “You are here” indicators is that they’re too subtle.

Search. Every site needs it regardless of whether or not it makes absolute sense. Everybody finds things in different ways, and many are wired to look for a search field by default. It doesn’t need to be anything grand; a simple text search of your site will suffice in many cases. No need to make it keyword driven, which adds to maintenance.

Designing a Home page inevitably involves compromise.

Don’t confuse a tagline with a motto.

All Web users are unique, and all Web use is basically idiosyncratic.

Testing one user is 100 percent better than testing none. In reference to usability testing.

The best-kept secret of usability testing is the extent to which it doesn’t much matter who you test.

In general, if the user’s second guess about where to find things is always right, that’s good enough.

I [as a user] should never have to think about formatting data.

Ignore all comments that users make about colors during a user test, unless three out of four people use a word like “puke” to describe the color scheme.