Thursday, April 2, 2009

Results of the All.html Test

The much awaited results of the all.html redesign test are here!
What were we trying to do?:(You can skip to the Results section if you already know about this ;))
In September of last year, I started looking at the current all.html page to see if there were ways to optimize the page. We had a high bounce rate (40%) and I felt that the page was visually confusing and that there was a better way to display the information listed.

I then asked The Royal Order (an amazing design firm) to create three designs to address the problem of trying to display 60+ Firefox localizations and their platforms to a worldwide community of people with different needs (ie, language spoken, ability to read English, various levels of tech-savvy, country of download, etc).

These were the designs we received:

All is the current all.html page. (This picture is of the older page because we had already taken the all.html page out of circulation by publication).
All-1 was our most "designed" of the pages that included a map. ((This picture is of the psd because we had already taken the all-1.html page out of circulation by publication).
All-2 was in the middle, with some geographical images and expandable lists.(This picture is of psd because we had already taken the all-2.html page out of circulation by publication).
All-3 was the most like the current all.html with some changes to some details like # of columns.

After looking at them, we were unable to decide on face value which one should be the new replacement. We all had plenty of opinions, but nothing based on verifiable data. Consequently, we decided that we would test all three designs (and the current design) to see which one performed the best.

The Test:We set up a week long test that ran from March 21-29, 2009. We rotated the pages evenly so each page showed for 15 minutes of every hour for the entire period. We defined success or "best performance" as:
1) Best User experience on the page comparing to all.html
2) Best download rate on the page comparing to all.html
(You can read more details about the test/process here and here.)

The Results:The results are organized by version of all#.html and broken up into page views and downloads. The conversion rate was found by dividing download clicks by page views. The baseline we used for all comparison was the all.html page. The baseline, therefore, is that 52.7% of people who hit that page downloaded a version of Firefox. Better Performance would then be defined as anything above the average of 52.7%.

All-1.html performed much worse than expected, with 32.4% of people downloading Firefox after hitting that page. All-2.html, performed at 56.5% which was on par with my expectations.
Lastly, All-3.html was comparatively stellar, getting about 60.9% of the people who hit that page to download. This basically translates to an 8% improvement over our current all.html page.
Now, before we get TOO excited about these results, resident Metrics Guru, Ken Kovash, asked me to do some further digging. Basically, he was concerned because "download clicks" represent a snap shot of a site visitor’s experience at a single moment in time and it doesn’t factor in what actions a person takes after that moment (e.g., a download occurring when that person subsequently navigates away to other pages within Mozilla.com).A Firefox user could hit the all.html page, then move on to the mozilla.com homepage, and then click download.In this example, the action of a download is not counted in our numbers and results above.

To figure this out, I looked at a funnel report in our web analytics system for all-3.html and all.html to try and figure out where people were going after the all.html pages to download (if not from the actual page) and to integrate these numbers into our results above.

All.html

First column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting colunms 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 67.6% conversion rate.

All-3.htmlFirst column you see below is total number of page views.
Second column is the total of downloads from the all.html download buttons (all of them).
Third columns is the total of downloads from the entry page (mozilla.com)
Fourth column is the total of downloads from download buttons that are not on one of our top pages (eg customize).
Fifth column is total after subtracting columns 2, 3, 4.
Sixth column is the number on the total downloads/page views which results in an average of 68.4% conversion rate.
Seventh column is the difference in conversion rates, with the average difference between all-3 and all.html being .89% (in favor of all-3.html).

SynopsisAll in all (pun intended) all-3.html was the best performer of the four pages. The 8% difference in user experience and .89% improvement in download rate (comparing to all.html) shows that even a small change in a page's design can have an effect on download numbers. And even though 8% improvement in ease of use or a .89% improvement in downloads may seem small, considering the number of people who see this site every day (70-90K page views a day) the effort and hardwork that went into this redesign and test were well worth it.

All-3.html has now taken the place of the old page and I'm looking forward to tweaking and testing some details and re-checking its' performance in the next few months.

Thanks to everyone who has put in so much hardwork these past two quarters: The Royal Order, Steven Garrity (Silver Orange), Wil Clouser, Jeremy Orem, Stephen Donner, Krupa Raj, Ken Kovash, Seth Bindernagel, Mike Morgan and John Slater--couldn't have finished this project without you!

I think the best thing to say is "We told you!", as most of the commenters said in the bugs. maybe next time a download page redesign survey would be enough. :)

Some thoughts: a. How many of these users actually got the map page, than navigated to another page and when they came back to the download page they got another design? It might be possible that some people that was unable to locate the download in the first two designs found how to complete the download process only for another design, and the rise in the download count for the 3rd design was due to bad download rates on the 2nd design...

b. When I investigated the download rates on our local download page, it come to my attention that some people are clicking twice on the download link, fast enough to treat it as a double-click, while the analytic software we use (Google Analytics) treat it as two separated clicks. Checking what are their current browser told me that most of them are IE6 users who probably just heard about Mozilla Firefox.

We have two different Firefox download pages on our community sites in Israel, and while the download rates are much smaller, I would be happy to share with you some of our stats which may give some more knowledge about how the users are finding the download buttons useful -

A. mozilla.org.il/firefox - Our main Firefox page, with a download links for Windows/Linux/Mac versions. The page is fully-loaded with texts about the browser, and it is somewhat complicated to locate the download buttons even if you can read the texts. 53% of the non-Firefox users are actually downloading the browser, and 5% are clicking on the link to all.html, probably in order to download the browser in a different language (English).

B. firefox.co.il/download - Our second download page, which is less known but useful for people guessing they can locate Firefox if they'll type firefox and the country tld (.co.il) after (most clicks are direct). We don't have text on that page, and 75% are actually downloading the browser from that page (they can select which language they prefer from the dropdown language selection there, and we are not tracking their choices).

C. Some users are actually downloading the Hebrew localized version of Firefox from http://www.mozilla.com/he. I don't have access to downloads stats from that page, and I assume we have similar rates as our local sites.

I don’t know If I said it already, but this so good stuff keep up the good work. I read a lot of blogs on a daily basis and for the most part Belajar seo blogspot just wanted to make a quick comment to say I’m glad I found your blog. Thanks.

Nice work and thanks!RunningAdidas currently manufactures several running shoesNike shoes, including the adiStar Control 5, the adiStar RideCheap nike shoesDiscount nike shoesthe Supernova Sequence and the Supernova Cushion 7, among others. Nike shox r4nike shox shoesAdidas also uses kangaroo leather to make their more expensive shoes. Association footballOne of the main focuses of Adidas is football kit and associated equipment. puma catcheap sport shoesAdidas also provides apparel and equipment for all teams in Major League Soccer. Adidas remain a major company in the supply of team kits for international football teams. cheap nike shoxcheap nike maxAdidas also makes referee kits that are used in international competition and by many countries and leagues in the world. In the United States, referees wear the Adidas kits in MLS matches even though the primary referee supplier is Official Sports. nike tn dollarnike running shoesThe company has been an innovator in the area of footwear for the sport with notable examples including development of the Copa Mondial moulded boot on firm dry pitches for forty years. nike air max tnpuma shoesAdidas became renowned for advancing the "Predator" boot design.This design featured a ribbed rubber structure for the upper leather of the shoe, used to accent the movement of the ball.discount puma shoespuma mens shoesThe Predator also features the Craig Johnston invented "Traxion" sole. As the development and popularity of Football continued Adidas played a leading role in shaping the style of the play itself. puma running shoespuma shoesFIFA, the sports governing body, commissioned specially designed footballs for use in its own World Cup tournaments to favour more attacking play. ghd hair straighteners mk4hair straightnersghd iv styler hair straightener ghd hair straightnerscheap ghd hair straighteners

Clothing has always been a thing that has been given a great importance by human beings. It displays the attitude that the people exhibit. ed hardy clothesed hardy shirtsLots of brands have been existent in the clothing industry and a famous one among them is the Ed hardy clothing brand.ed hardy jacketsed hardy hoodiesThe brand got its name from the famous American tattoo artist Ed Hardy. He was a very famous tattoo artist and has published many books on tattooing techniques. ed hardy bootsed hardy polo shirtsBut his tattooing turned into a brand by the efforts of a company called Christian Audiger. This company was a very famous and very powerful company in the field of clothing. ed hardy shoesed hardy jeansThey felt that it would be appropriate to create a brand called Ed hardy and use Hardy's art as the main selling point for the brand. This venture had turned out to be a very successful one and Ed hardy clothing is one of the most famous brands in the clothing industry.ed hardy outerweared hardy long sleeve shirtsThe brand became very famous because it was worn by many famous celebrities like Madonna, Britney spears and also Sylvester Stallone. ed hardy bagsed hardy winter bootsThe brand has clothes for men, women, and kids. They have also diversified their business by having a lot of accessories to support their clothing business. ed hardy love kills slowly shirtsed hardy love kills slowly shoesed hardy love kills slowly bootsfor men they have a variety of products such as active wear, denim, outwear, swim trucks, sweaters, t-shirts, tops etc. ed hardy trousersed hardy mensthe accessories include things such as belts, caps, scarves, shoes, socks, jeweler, sunglasses, ties and even wallets.The art works is a unique combination of American and Japanese cultures. The unique selling proposition of the brand is the way in which the company has used the art works of Ed hardy.ed hardy womens ed hardy t shirtsed hardy sunglasses

thanks my dear.. this is great article for me. I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading.

Although everyone Biber Hapı knew exactly what they want from these words can be attached to. If you Biber Hapı think you can participate in a good way.Added an idea what it would have the power to make you guess. I recommend it Biber Hapı to you in any way.As we always Fx15 say we're right on this issue and will protect you from Fx15 various complaints by developing a system that he will run. We are Fx15 always wishes us to continue to monitor, get better.

Many thanks for your themed, a good post for me.Now it will be very cold in winter, in order to keep warm yourself, moncler company has provide jackets for you, so it is better to own moncler yourselif as soon as possible. Eeveryone love fashion clothing, polo ralph lauren is very popular all over world, that is my dream to get one.We know Ray Ban by America soldier, all of them wear ray ban sunglasses when they walk on the road, it is fashionable for you. Most of people like to wear jeans, it is very modern when wearing true religion jeans in the street.

The fungus is another B vitamin pantothenic acid, especially in times of stress the Biber Hapı adrenal glands helps protect fatig'den supporting the operation. Showing the great importance of this effect of zinc.

Hello to everyone who has blog.first I congratulate the blog owner.he is got excellent blog.That is really very good article. I am glad to know. Thanks for sharing !kizlarla chatcetavrupa chatchat odalariMersin Chatsohbet odalari

Yemek Tarifleri Thanks for this post – I have struggled with this as I have NO comments yet on my blog and in some ways feel badly about that (until i read your posts about this) and yet I see there is more that I can do. thankss.

hey i do love your post and this blog make me awesome..very useful and i like to share this into my friends just to read your post...well this blog sayin keep up your dream and lets make that together..thanks again for posting..what a great blog.

Some thoughts:a. How many of these users actually got the map page, than navigated to another page and when they came back to the download page they got another design? It might be possible that some people that was unable to locate the download in the first two designs found how to complete the download process only for another design, and the rise in the download count for the 3rd design was due to bad download rates on the 2nd design...clenbuterol | steroids for sale | steroids

What a Great Article it its really informative and innovative keep us posted with new updates.its was really valuable.I must admit it, terrific job on this blog,I’ll be sure to check back again real soon.