Testing and Coding…Concurrently

Category “Mozilla”

Even though the software credo I am writing is a personal thing, I’m not writing it in a vacuum. We are all writing the history of software and, at this point, the history of computers and software is big enough and old enough to have it’s own corners and back alleys.

In this post, I’ve researched into some questions about computer & software history. I’ll be writing about some events that were important in my corner of computers, some of moments which were not the best, and the event I would most like to have witnessed.Who are some of the important people or events in your particular area of software and what did they contribute.

I’ve already blogged about The Ultimate Nerd and my ultimate nerds so I’ll be focusing mainly on the events in computer and software history that has meant the most to me.

The fight between Internet Explorer and Mozilla
I may have just left Mozilla as a corporate employee, but Mozilla and its mission are still very much alive to me. If you don’t understand what the whole fight for the open web is about, it is worth Investing 40 minutes of your time to watch Mitchell Baker talk about the history of Mozilla.

Back in the nineties, I remember listening to NPR every day for news about how the lawsuit between Microsoft and Mozilla was proceeding. I hope that, at some point, a book is written about the history of Mozilla and some of its projects. I had chills more than once as I watched Mitchell Baker give this talk on the history of Mozilla. A lesson she learned from the Mozilla project and her most memorable quote from this talk is something I will carry around with me until I die, “Leadership depends on who will follow you.” (It’s at 11:30 if you wish to listen for yourself)

The blossoming of the open source software movement
The theology of the open web and open source software is deep water which I’m not expecting to plumb in a couple of paragraphs, but if you give yourself the time to really dive into the history and its ideas, you will be rewarded.

If you wish to wade into these waters, I highly recommend reading through The Cathedral and The Bazaar by Eric Raymond. It is beautifully written and I think I must have highlighted half of it. Although there are frequent references to the creation of Linux, the paper itself is timeless like K&R or Unix Shell Scripting.

Reading through this paper, I could see some of the groundwork for agile being laid. There is a spirit of egalitarianism coupled with a “need for speed.” Raymond mentions in a few places that it is important to “release early, release often.” He also writes about the very inclusive development philosophy of Linus Torvalds which was counter to the more exclusive “cathedral” model of isolating a few geniuses and letting them polish the software creating a longer release cycle.

This actually deserves a longer post and critique in the context of what we know about open source today.

Looking at the number of people who were present for this event, I will never understand how they were all able to agree on the document itself. It appears to me to be one of the greater examples of consensus. The fact that what’s in the manifesto meant enough to these guys to get together and agree on it sends a strong message. I consider myself lucky to swim in this every day at Pivotal Labs and I hope my blog helps you push further with it in your own professional life.

Historical Software Defects
These are the moments in software history that are not the greatest but they have valuable lessons to teach.

Therac-25
The, “primary reason should be attributed to the bad software design and development practices, and not explicitly to several coding errors that were found. In particular, the software was designed so that it was realistically impossible to test it in a clean automated way.”

The Therac 25 was designed to automate the delivery of radiation therapy to cancer patients. Tragically, it sometimes injected patients with levels that were too high, even tragically high.

My software engineering teacher, Dr. Susan Duggins, first introduced me to this in our software engineering class. It’s important because it highlights that testing should be involved earlier in the software process and that building software is not just about typing out the code. I am in love with the idea of ecosystems as they apply in software and in open source. This legal case points the way towards software occurring in an ecosystem.

The Mars Rover
Imagine that you’ve spent months working on a small vehicle that will land on Mars. Imagine the pressure of knowing that many millions of dollars has been spent for you to do this work. It’s a crowning achievement involving your team and other teams as well.

Imagine that the Rover lands and doesn’t work because you’ve been programming in standard measurement units, but an external team you’ve been working with has used the metric system. I would have cried for days.

The failing of the Mars Rover demonstrates the power of good communication and how major defects can occur without it. If you are a tester and you sometimes feel like the team therapist because you’re trying to get developers to talk with each other, I have news: You are not alone in feeling like a therapist. If you ever wonder if you are doing the right thing or sticking your nose where it doesn’t belong, think of the Mars rover team.

If you could only witness one moment in computer or software history, what would it be and why?
For this question I am brazenly cheating. I’d like to watch one of the great visualizations being drawn just to see the tools that were used and the place where it was being drawn. These had to be hand drawn as there was no machinery to produce them. I’d like have a look at the instruments used to make the measurements and the drawing implements. I’d like sit in the chair that William Playfair sat in or watch Charles Joseph Minard explain his visualization of Napoleon’s March.

This concludes my look at software history for my blog post, but it’s brought up some threads I’d like to push further. I’m not quite finished reading and writing about “The Cathedral and the Bazaar.” I’m also not finished with “Leadership depends on who will follow you.” It’s a funny thing about these credo posts. They tend to open more doors and windows than I have time to close. I don’t mind leaving them open, however, as this is letting in some fresh air. Wherever you are, I hope that when you get to the end of my post, you take a minute and just…breathe.

As a tester of web applications, I’ve become familiar with how web pages are constructed and rendered through the browser. Firefox Add-ons provide a great set of tools for doing this and have been a mainstay of my testers toolbox for quite a while. One of my jobs as a Mozilla employee was to show others how to use some of these addons.

Below is a chat I had a while back with a contributor (name anonymized for privacy) about Firebug and Firepath, 2 addons that I use all the time for getting information about the elements, called locators, on a webpage. We use these all the time in Selenium testing and it is a taste of what I’ll be talking about in a free webinar I’ll be giving for the Software Association of Oregon on Tuesday, May 15 from 12:00pm to 1:00pm. Registration is free :o)

In addition to Firebug and Firepath, I’ll be talking about an add-on developed at Mozilla, Mem Chaser and some browser functionality that started as an add-on but has made its way into the Firefox browser, Tilt.

marlenac Have you used Firebug before?olivier No i have heard of itolivier Its a firefox extensionmarlenac You’ll want to install 2 addons: Firebug and Firepath into firefoxmarlenac I use both of these all the time to help with the locators. I’m also looking up a good link on CSS for you.marlenac This blog post explains some of it and has a bunch of other links for working with Selenium and CSS: http://blog.reallysimplethoughts.com/2010/10/12/a-quick-introduction-to-css-locators-in-selenium/olivier How does firebug help with locators?olivier Does it generate the expression?marlenac It allows you to inspect web pages so you can see what the locators are.marlenac If you’ve got it installed, choose an element on a web-page, right click and choose “inspect element”olivier Kkmarlenac As an example, I’m on the homepage for addons: https://addons-dev.allizom.org/en-US/firefox/marlenac If I hover the mouse over the big “ADD-ONS”, right click, and choose inspect elementmarlenac It will split the window of the browser and show me the html of the page at the bottom.olivier Kkmarlenac the line starting with “<a title=”Return to the Firefox…” is highlighted.marlenac Just above that is a css class “site-title”olivier Rightmarlenac If I want to select the link at “ADD-ONS”, the selector will be “.site-title > a”marlenac We can check that this is correct with Firepath.marlenac In the Firebug pane, there are several tabs at the top. “HTML”, “CSS” and further over is “Firepath”olivier Kkmarlenac If you paste in “.site-title > a” without the quotes, it will highlight the element for you.marlenac It’s pretty great!olivier Thx so much u just made my life easiermarlenac I know!!!!!! We’d all be suffering without Firebug and Firepath!!marlenac A couple of tips.marlenac Use .blah to specify a class namemarlenac Use #blah to specify an id.marlenac Use the “>” to get to a child element.olivier What if there are many child element?marlenac You can keep going with it.marlenac or if there is a list, you can use “nth” to specify the 1st, 2nd, 3rd.marlenac This post has a great explanation of that: http://saucelabs.com/blog/index.php/2010/01/selenium-totw-css-selectors-in-selenium-demystified/marlenac I’m just finding an example in our code as well.olivier I think i get itmarlenac Cool!olivier I will ask i get stuckolivier Thx againmarlenac You’re welcome :)

In a few weeks, I’ll be joining Pivotal Labs to work on the Pivotal Tracker team. I’ll be mainly handling support requests and helping with some testing as well.

What drew me to Pivotal? After all, I’ve got a job as a Software Engineer in Test at Mozilla working with Selenium tests all day every day, what more could one possibly want? Judging by the number of recruiter emails I get, a Software Engineer in Test working with Selenium all day can pretty much right their own ticket, can’t they?

Well, yeah, and I <3 Selenium, but it’s just one part of testing. In fact, writing Selenium tests is just one aspect of making software. I’m ready to own up to being a specialist who knows a lot about testing and automation, but I’m also a generalist who helps make software. At one point, I thought I only wanted to work on test automation infrastructure, but I’ve since learned that I prefer working with a product team.

This all came about while writing the credo posts that have pre-occupied me since January. I’ve learned that I love writing more than any other occupation and that participating on a team making software is more important to me than identifying as a tester.

This change will move me into a world where I toss out my own self-imposed label of “tester” or “automator” and throw my bucket of skills at a highly collaborative software team. In letting go of being “the tester,” there will be other skills that I now get to exercise:

Being a great teammate. While this is important at Mozilla, I expect even more emphasis on this in the tightly coupled, rabidly agile environment of Pivotal Labs.

Since Tracker support is 100% email (plus whatever y’all throw me on twitter), I get to use my communication & writing skills as a primary part of my job.

The x-factor skill which isn’t an obvious skill in software, but will be crucial for support is having a developed sense of “mindfulness” or non-judgmental awareness.

This is all in addition to flexing my technical skills at bug isolation.

Despite leaving Mozilla, I still have quite a fondness for the company, its mission and my teammates there. As a parting gift to them, I have stolen the following anonymized excerpt from deep within the bowels of irc. I hope it gives you a chuckle and some encouragement to, “whisper to the fox:“

FirefoxLvr404: come to think of it sweatsbac0n, I’ve seen you blogging, but I haven’t seen you blogging about how opensource html5 makes you opensmile

FirefoxLvr404: how much more delightful canvas web technology do you fucking need?

Last week, I isolated a performance problem that started with the failure of an automated test. I’m blogging it because the bug has an interesting story which highlights some of the weirdness I’ve typically found when isolating performance bugs.

A teammate of mine, Alin Trif, noticed one of our automated checks failing. It writes a review for an add-on. He replicated the failing test manually and wrote up the bug. Since a few others in our group couldn’t replicate the problem, it was subsequently closed. In true tester fashion, Alin mentioned it to me anyway. I thought “what could it hurt to try reproducing it.” So, after opening our staging site, addons-dev.allizom.org, I went through the steps…and reproduced the problem.

Note that I did not just re-open the bug. Personally, I think re-opening bugs is a great way to alienate people who are ready to move on from a particular issue and make testing more irrelevant on your team. This doesn’t mean dropping the ball on a problem. It means that it’s time to uncover more or better information which would likely result in a different bug being opened anyway. If you read to the bottom, you’ll find that this was, indeed, something else entirely.

Two of my favorite web testing tools are Firebug and Skitch. Although I generally use Firebug for inspecting web pages and finding the locators I need for writing Selenium-Webdriver checks, it will also show metrics for requests and responses. To access this, open Firebug and click on “net”

Firebug -> Net

In this case, while the request was off in never-never land, I opened Firebug, switched to the net tab and took a snapshot. When the request eventually finished, I took another snapshot. This time, the culprit is pretty clear.

Although I was testing the addons staging site, the bug is actually for Browser-ID, Mozilla’s new solution for single sign-on.

Unless you are actively looking for performance bugs, they are extremely easy to dismiss. In this case, it looked like it wasn’t reproducible. These are also easy to miss unless you are hyper-sensitive to slowness. (Every tester ought to know their physical triggers of impatience for slowness. Do you tap your fingers and/or roll your eyes and/or cuss at the screen?)

If you’re frustrated about one of your bugs getting closed or marked invalid, it’s time to talk to someone about it vs. only leaving a comment.

Using Firebug and your screenshot tool of choice makes it fairly painless to document these bugs. I’ve caught bugs like this before and ended up installing a third party tool for looking at the time taken for requests and responses. This is now much easier if you download the Firebug add-on because the tool is right in your browser.

The next time, something “feels slow” or “slower” to you, give this a try. You might find something you weren’t expecting.

Hat tip to Alin Trif for finding the bug and asking about it even after it was initially closed. That’s good testing.

Today, I’m blogging from the HTML5Dev Conference. The house is packed, and I’m breaking out of my shell as a tester to have a look at all of this from the dev perspective. Of course, the tester in me has tagged along. What’s great about conferences in the age of twitter is that you are never in a bubble during these things. One of my co-workers, Greg Koberger tweeted about some performance testing win for the lastest Firefox release, Firefox 7.

Greg's tweet

In taking a look at the lifehacker article he tweeted, I noticed that there was a category for css performance tests. Since I spend all day, every day looking at selenium tests that have css locators and I sit next to a css dev (who has excellent taste in rap), css has been on the brain lately. “Hmm…css peformance…LET’S GOOGLE.” If you google for css peformance, the first link is for an article titled, Performance Impact of CSS Selectors, by Steve Souders. Given, the article is talking about Firefox 3.0 so it’s obviously a bit long in the tooth, but it’s interests me for not one, but two reasons:

I’m waiting for the talk “High Performance HTML5″ to begin. It’s being given by Steve Souders. (Serendipity or Coincidence? I shall leave you, dear reader, to ponder.)

The article has a list of different websites and the number of DOM elements in each one.

Since this is a guerilla blog post I’m finishing up as a talk starts, I have no intention of delving deeply into the guts of the article, at the moment. What I can share, however, is a new Firefox addon I’ve been playing with called Tilt. The picture you see above is my Facebook page, turned on it’s side in tilt. Tilt visualizes a web page’s dom elements in 3d and was developed by a Mozilla intern who has recently been hired. So if we filter the list in the article, Google has the least number of elements while Facebook has the most. It should be no surprise that the Google home page has had its performance tweaked to oblivion. Here we can compare the dom of Google visually with the Dom of Face book.

Facebook:

Facebook Tilt

Google:

Google tilt

I’m definitely forming a hypothesis about the CSS performance of these two pages based on their tilt results.

And that’s your guerilla blog post from the HTML5Dev Conf. They really ought to make this thing 2 days next year. It’s pretty cool.

A phrase I hear a lot around Mozilla is “continuous deployment.” I hear there’s this product Mozilla makes that’s competing with some other product that has rapid release cycles. So, yeah, we’re working on continuous deployment.

I’ve noticed that a main resource around our office for information about continuous deployment is this video from Etsy. Hearing, “We’re moving to continuous deployment,” is nothing new for me. This is the 2nd job I’ve had where it’s been a major focus. Since I’ve heard of the Flickr version, I decided to watch this Etsy video.

Picture yourself at your computer about to hit the big button and deploy a feature you’ve been working on. You are fairly confident that nothing catastrophic will happen, but you don’t know. (I’m writing this from a dev perspective, but even if you’re a tester…come on…you never know, even if you’ve tested the hell out of something). In the talk, this is what is referred to frequently as, “the fear.” It’s actually referred to as either, “THE FEAR” or “the fear.”

“Fear for startups is the biggest no-no.”

“Fear is what keeps you from deleting your database.”

“Fear doesn’t go with creative work.”

This rings true for me because I frequently deploy selenium tests for addons.mozilla.org. My teammates and I have talked about “THE FEAR.” We have strategies for coping with it such as holding one’s breath, saying a prayer or running the 90+ tests one more time. When Etsy talks about “The Fear” I know exactly what they mean.

Etsy’s video fascinates me because of how they have conquered “The Fear.” It’s been on my mind every day since I watched the video. What’s the special-continuous-deployment-sekrit-sauce-that-makes-everything-all-better?

Etsy combats “the fear” with visibility. You see, at Etsy, EVERYTHING IS GRAPHED ALL THE TIME.

Here are some of the things they mentioned graphing in the video:
How many visitors are using this thing?
Can we deploy that to 100%?
Did we make it faster?
Did I just break something?
How long is it taking to generate a page?
How many users are logged in?
How is the bandwidth?
What’s the database load?
What’s the requests per second?

If you look at the graphs, they are simple bar or line graphs. They are not exceptionally fancy but they are numerous and the maintenance admittedly takes work. They are not, however maintained by specialists working in a silo. The graphs are created by an engineer. Here are some numbers:

20,000 lines/second is their log traffic, at times
16,000 is the number of “metrics” they have organized through dashboards
25 engineers committing code to dashboards
20 dashboards

I doubt that when Etsy decided to start graphing everything they woke up one day with 25 dashboards. It sounded very much like they put the tools in the developers hands and lovingly nudged them along.

This is a serious commitment to data.
Data doesn’t just happen. It takes a persistent effort to include log messages in your code. It takes servers and databases capable of handling the traffic created by the log messages and staff to maintain them. It takes investing in huge monitors all around the office and giving people the bandwidth to figure out how to work with the data & graphics stack. Most importantly, it takes trust so that employees are allowed to see the data without making them jump through hoops.

So how can a team move closer to the graphing part of continuous deployment?
According to Etsy:

Give people access to production data — without making them wait months for a special password or even log in every time.

Make the data real time instead of daily. When I say access, I mean feeds. This goes well beyond a spreadsheet.

once you have the data, make graphs for features before you release them

I love data, but will be the first to admit that it is not pretty. The plain truth about data is that it takes patience because combing through and refining it can be tedious, monotonous work. It is very easy to buy a bunch of monitors and put them on a wall showing an inst-o-matic graph that came with your bug tracker (I’ve seen this done. O hai, expensive wallpaper!). It takes more time to ask deeper, meaningful questions. It takes even more time to filter the data into something graph-able. After that, you have to find the right way to share it. Note, that even if you do all of this and the data successfully tells a story, you’ll have to spend time dealing with, “and why did you use those colors.” What was I saying? Oh yes, data is not pretty.

Now that I’m working every day with tests I visualized a couple of years ago, I’m continuing my quest for deeper questions about tests. In my context, the tests are the selenium tests I work with day in and day out, so besides coming to grips with “THE FEAR,” I’ve also been thinking about, “THE FAIL.” But wait! That’s another blog post.

Next Friday will be my 2nd AMO Automation testday at Mozilla and the first one I’ve helped to organize. AMO stands for addons.mozilla.org and it’s the web-site for which I’ve been writing and reviewing automated checks (sometimes I call them tests, but I like referring to them as checks).

Grid by msmail

As part of the last test day, I ran a github workshop to help people figure out how to set up and work with github. Here’s a link to that blogpost.

For the upcoming testday, I’ll be running a workshop to help people who would like to run our addons selenium checks with our grid configuration. Although it’s easier, when starting out to write and execute tests using a standalone selenium jar, it’s good to understand how to run tests with selenium grid as well. All of the tests for AMO are run using grid, so this is the setup I use when I’m doing code review or writing checks.
This is the link to the github repo containing our grid configuration. I’ll be posting again next week with a few more instructions on how to modify the repo for running checks locally.

The testday starts at UTC/GMT 15:00:00 and continues throughout most of the day until 5:00pm Pacific time. (We’re global, baby!)

A few local times for the moz-grid-config workshop:
UTC/GMT: 17:00:00
Pacific: 10:00am
London: 6:00pm
Bucharest: 8:00pm
Bangalore: 10:30pm