Tag Archives: Mozilla

Building software is fun. Spending countless hours or even days on something to get it finally working. Helping someone who uses your software to speed-up the daily workflow. All that is fantastic and every developer loves that. But don’t let the dark side come up, when customers are pointing you to a lot of software defects. Are you still proud and will you continue the work as how you did it before?

Most likely not. Or well, lets say at least not when quality is what you want to ship. So you will start to think about how to test your application. You can do a lot of manual testing based on test plans or just do exploratory testing. That will work as long as your application is not complex enough, and can be done in a couple of minutes. But once you have to do the same kind of steps over and over again for each release, you will find it boring and loose the interest or concentration on it. Failures will happen so that issues slip through your testing, and bugs becoming part of the new release.

That’s indeed something you eventually want to kill in the future. But how? There is an easy answer to this question! Use test automation! Create tests for each feature you implement, or regression you got fixed. Over time the enlarged suite of tests will make you happy, given that you have to spend nearly no time on manual tests, and have results in a split of the time needed before. Releasing new versions of the application can be done much faster.

At some point, when your application is large enough, you might even not work alone anymore on that product. There will be other developers, or even software testers whose job is to plan and execute the testing strategy. Given that in the past there was not such a high demand on automation knowledge for them, the requirements for jobs have been changed in the past months. So lesser companies will hire engineers for quality assurance who do not have a coding background. This is hard for them, given that it can take ages to find a new position. Something has to change for them.

We, the Firefox Automation team at Mozilla want to help out here. Given our knowledge in automation for various Mozilla related projects, our goal is to support interested people in gaining their knowledge in software development and especially test automation. Therefor we are planning to have automation trainings on a regular basis. And all based on our own projects, so you will have the chance to practice all the new things, which you have learned. All that indeed depends on the acceptance for that offer, and the number of participants.

The first two training days will happen on March 24th and 26th, and will mainly take place in our #automation channel on IRC. Given that we have no idea how many of you will join us during that day, and what your knowledge is, we will start with the basics. That means we will guide you through courses of Javascript, Python, HTML, or CSS. We will collect your feedback and extend the etherpad for automation trainings to finally have a wonderful list of getting started tutorials.

For those of you who already have more experience, we will offer tasks to work on depending on your skills and directions. Please see the before mentioned etherpad for areas of work and appropriate mentors. We will guarantee that it will be fun!

We would love to see you next week, and if you have questions, don’t hesitate to ask here, or in the automation mailing list.

Our last week was again very productive, and a lot of smaller and larger tasks have been finished. We are getting closer to reach our quarterly goals. Our next big step is really to get the final version of Mozmill 2.0 out, and the Metro support finished. Then we can get started with implementing ui driven tests as requested by Mozilla QA.

Hightlights

With the combined work with our Softvision team and a couple of contributors we were able to release two versions of Mozmill in one go. See my blog post for release details of Mozmill 1.5.22 and 2.0rc5. Hopefully the 1.5.22 release will mark the end of life for the 1.5 branch, and that the 2.0rc5 was the last release candidate build for Mozmill 2.0. More to that topic soon.

To be able to move our Mozmill l10n tests from an older Mac Mini, which is currently running in the MV network, to the more stable and IT supported network in the SCL3 datacenter, we had to fix a couple of issues in mozdownload. So as result we have released mozdownload 1.9 last week. That fixed the last remaining blocker, and we will have the l10n tests running on staging soon.

Over the last couple of weeks we were monitoring the memory usage of Jenkins a lot. That happened not only in the browser but also more important for the whole system on the server side too. As we noticed the Java process has a really high heap memory usage. Beside that we had a couple of annoying issues around, which caused some of our testruns to early abort. So Henrik put a lot of work in getting Jenkins upgraded to the latest LTS (Long Term Support) version. Given that this was a big jump for our really outdated Jenkins version a lot of work had to be spent in testing everything. Also because nearly all of the Jenkins plugins, which we make use of, had updates available and had a slightly different format of the XML configuration, all our static files had to be updated too. But by early last week we were able to push those bits to our staging server, and running that version of Jenkins for a week, hasn’t brought up any kind of regression. That’s great! To finish up the work, Henrik published all bits and bytes to our production machine yesterday. We will continue observing the memory usage, and let you know if the situation got better.

Beside that we made a couple of changes to our list of available nodes for the mozmill-ci cluster. As of last week we brought our Windows 8.1 nodes (3x 32bit, 3x 64bit) online for preliminary testing. Given that all went well and no further issues have been seen, they are now fully integrated into jobs triggered by Pulse notifications. Further we have replaced our existent Ubuntu 12.10 nodes with the current release of Ubuntu 13.04. That means we run our tests on Ubuntu 12.04 (LTS) and Ubuntu 13.04 for both 32bit and 64bit now.

As requested by Mozilla QA we have to run more tests for localized beta and release candidate builds of Firefox. That will help us to finally stop releasing broken l10n builds like Firefox 24.0b1 to our users. For all locales covered by our tests, we do now check for start-up failures like the XML parsing error for the main window. So Henrik pushed a patch to production, which enabled tests to be run for the first 10 most used locales of Firefox.

Oh my! It happened again. And here it is. Another release of mozdownload with a couple of new features included. As a regular user please let us know if all is working well, or if you want to see some new features included.

Below you can find an overview about the main features in mozdownload 1.9:

Starting from today, the Automation Development team wants to publish weekly reports about the work happened in the previous week. Reason is that similar to other groups we want to offer everyone more insight into our projects, and that we might find even more community members who are interested to help us and learn new stuff at the same time. If you have comments or proposals to the style and contents please let us know.

Highlights

Beginning of last week we promoted Andrei Eftimie, who is working for our team as a Softvision contractor from Romania, as peer for our Mozmill Tests repository. That happens in regard of all his contribution in the last couple of months. So if you are looking for a reviewer of your patch, you can flag him from now on. Congratulations again.

Dave Hunt got the performance tests running on the “Leo” devices now. Those are testing the cold load times and fps of the applications under test.

Over the past couple of weeks we detected a lot of application disconnects while running the Mozmill tests for Firefox under the Mozmill-CI system. As it has been turned out notifications (bubbles) from application updates could be the problem, and especially the new update overlay screen for Windows 8 updates. The latter completely blocked our testing. So Henrik Skupin re-configured all existing nodes and templates by disabling any automatic update checks (Bug 900860). Since that has been done, no further application disconnects have been observed. Lets cross the fingers that this will stay that way!

We noticed a massive disconnect of slave nodes in our Mozmill-CI cluster early last week. At the same time the response time of our Jenkins master was drastically slow. Henrik investigated the problem and identified that our current VM is not able to handle the memory load with 2GB of memory. So we increased the memory to 4GB, which indeed stopped the swapping process of consuming about 60% cpu load nearly all the time. Henrik will continue to watch the current memory consumption over the next weeks. Further we will upgrade our Jenkins installation to the latest LTS version soon, once all blockers are fixed.

Woot!!1! MozCamp EU 2012 is approaching really fast. Only two days left until a couple of hundred supporters of Mozilla will meet in Warsaw to plan and work on the future of the open web. It’s exciting to be part of it and to see you all again!

If you are interested in raising your skills in automation and want to help us to strengthen the quality of Firefox OS please join us on Saturday at 3:05pm in B2G room 1. We still have a couple of seats left. To be best prepared please read through our session details and try to setup your machine before you leave for MozCamp. We have setup links to get all necessary tools and the prepared Ubuntu VM. If you aren’t able to do that, don’t worry, we will have USB thumb drives available for you to grab all the data you will need.

As a sneak-peak we will also give you a link to our introduction slides so that you can change your mind if you are unsure about your attendance:

We are looking forward and can’t await to meet-up with you all. So see you @MozCamp on Saturday!

Update: We are planning to send out prizes for the best hand-crafted test(s) some days after MozCamp when we have reviewed all the written tests. So please be aware of code correctness, coverage, and style while implementing the tests.

It’s was clearly a wow! The way how our first “Ask an Expert” session went was promising. I haven’t thought that we would be able to fill a complete hour with discussions right away – but finally we ended up with additional 15 minutes to get the next steps set for Mike Kaply’s last question how to run automated tests for autoconf.

So what is that amazing on such a session? Given that in the past we as team were mostly working on Bugzilla to get a plan how to solve a specific problem, a live discussion is indeed a lot more helpful. Every attendee can directly participate, ask if there are unclear steps or if there are better ways to accomplish the wanted goal. The discussion is ongoing until you really met your expectation, or the time is up. That’s the difference to threaded conversations via email, where you should expect delays in the turnaround. You will lose valuable time which is even harder when you already have a strong timeline to finish up a feature.

The Automation Development team strongly believes that we all can work together to improve the Test Automation area at Mozilla even more. To achieve that goal we will continue in having live discussions each week and to help others in increasing their skills in automation.

And as a hint for all MozCamp attendees: If you are around keep an eye out for our “Make Sure Your Code Works” hacking session on Saturday at 3:05pm. I will post details about it before MozCamp but just as a heads-up so you can mark it in your calendar.

As you might have noticed in the last couple of months a group of people inside the Automation and Tools Team (A-Team) have been formed to help in making the automation expertise better at Mozilla. We are known as the Automation Development team and currently consists of Dave Hunt, Rob Wood, and myself. If you haven’t met us yet or even never heard our names, please check our team member section where we offer some more information about us and nice pictures. Better now?

At Mozilla we have dozen of test frameworks in use; like Mochitests, Selenium, Mozmill; which mostly dependents on the product under test. With this large matrix it’s sometimes not that easy to figure out when a test can be automated and which framework is best to use for. Even there can be overlaps because multiple frameworks would fit for a particular test. Confusion is pre-programmed.

So the Automation Development team wants to bring some light into the darkness by offering help for those questions. Therefore we will start to host office hour like sessions each week on Thursday at 12:00pm UTC. This will be done via our Vidyo infrastructure and you will only have to install the Vidyo desktop client as suggested when trying to enter the room the first time. For details and up-2-date information please visit our meeting page.

Given that we don’t exist that long yet and are 3 people only, we will start with the Mozmill, Selenium, and B2G testing frameworks. We will extend our coverage to other frameworks once we have the knowledge and man power, and know that sessions like those are helpful and accepted by others. Also please note that the current time has been chosen to start improving our relationship with community members in Europe first – here most of them are located for the above mentioned frameworks. In the future we are planning to have another session for US folks.

We are kinda excited about this opportunity and we would like to invite everyone who has interest to join and actively participate in the discussion.

Yesterday we had to release Mozmill Crowd 0.1.5 due to changes in our reporting infrastructure. Given the fallout of one of our servers our Mozmill Dashboard is now located at http://mozmill-crowd.blargon7.com/. This move needed an update of the default report URL. Now you can send reports of your test results again.

We have to admit that not much work happened on the extension in the last couple of months. Reasons are mostly other important projects and mainly the development of the new pre-configured environment for Mozmill. There are still some outstanding improvements to implement before we can release a new version of the environment.

If you are interested to help out please get in contact with us via our team page or any other public channel.

To help the team around the Memshrink project Mozilla QA Automation Services has released Mozmill Crowd 0.1.4. In this version the only changes which affect the extension itself are additions for the Endurance test-run, which allow users to specify the number of entities and if Firefox has to be restarted in-between each test.

We also have moved our Mozmill environments for all platforms from my own Mozilla people account to a new secure location on mozqa.com. In the future please always use this new location to grab the latest version of the environment.

For more information please see the tracking bug for Mozmill Crowd 0.1.4.

As part of the discussions during our QA Automation Services work week end of July, we have decided to be more open to the public. We have seen that in the past mostly all conversations regarding tools, tasks, and status updates happened on our internal mailing list, which is not accessible by any of our contributors. Well, that’s counter-productive because we have dozens of smart and awesome community members out there who probably want to help and learn the techniques in automated testing. And we should give those the possibility to easily participate.

As result we have raised bug 676224 two weeks ago with the request to create a public facing mailing list and newsgroup. The creation has been a bit delayed but finally happened last Friday. So if you want to participate in automation, feel free to join by subscribing yourself to one of the following communication channels: