Thursday, November 09, 2006

Marissa Mayer at Web 2.0

Google VP Marissa Mayer just spoke at the Web 2.0 Conference and offered tidbits on what Google has learned about speed, the user experience, and user satisfaction.

Marissa started with a story about a user test they did. They asked a group of Google searchers how many search results they wanted to see. Users asked for more, more than the ten results Google normally shows. More is more, they said.

So, Marissa ran an experiment where Google increased the number of search results to thirty. Traffic and revenue from Google searchers in the experimental group dropped by 20%.

Ouch. Why? Why, when users had asked for this, did they seem to hate it?

After a bit of looking, Marissa explained that they found an uncontrolled variable. The page with 10 results took .4 seconds to generate. The page with 30 results took .9 seconds.

Half a second delay caused a 20% drop in traffic. Half a second delay killed user satisfaction.

This conclusion may be surprising -- people notice a half second delay? -- but we had a similar experience at Amazon.com. In A/B tests, we tried delaying the page in increments of 100 milliseconds and found that even very small delays would result in substantial and costly drops in revenue.

Being fast really matters. As Marissa said in her talk, "Users really respond to speed."

Marissa went on to describe how they rolled out a new version of Google Maps that was lighter (in page size) and rendered much faster. Google Maps immediately saw substantial boosts in traffic and usage.

The lesson, Marissa said, is that speed matters. People do not like to wait. Do not make them.

Traffic and revenue from Google searchers in the experimental group dropped by 20%. Why, when users had asked for this, did they seem to hate it?

Since when does lower traffic and revenue FOR Google equal less satisfied customers? Did she mention anything about how satisfied the users actually were with the results?

If Google were omniscient and somehow returned exactly what I wanted *every* time I search, my traffic on Google would drop because I would be doing less searches. It seems like Marissa would interpret that to mean I'm not satisfied with Google's results, when in actuality, the results were ideal.

I guess this is the strange game you play when your company makes more money when people can't find what they are looking for as easily (and then click on ads).

> If Google were omniscient and somehow returned exactly what I wanted *every* time I search, my traffic on Google would drop because I would be doing less searches. It seems like Marissa would interpret that to mean I'm not satisfied with Google's results, when in actuality, the results were ideal.

I did a small test: I ran a Google search. 10 results, three ads. I than adjusted my preferences; 50 results, three ads. Speed difference? Imperceptible. Satisfaction? Much higher.

I rarely get what I want in 10 results, and having to wait for the next page is annoying. More results per page, with relatively fewer ads, is a win for me, but could reduce ad revenue.

Sites, such as Amazon, that seem to default to too few results per page make me think that the owner is trying to maximize ad exposure while not pissing me off so much that I'll leave.

Speed matter most when there are no other factors driving loyalty. But when a site offers quality, I'll be far more patient.

It may make sense for Google to default to 10 results, if the idea is to quickly impress new users enough to make them want to come back. The option to increase the number of results is good because, on average, it saves time.

I have to agree with the previous two comments. If traffic dropped in the experimental group, it would stand to reason that searchers were finding what they were looking for faster. And were therefore more satisfied.

Hey, if I was Google, I'd be like all, "More money for us = more satisfaction", too. But a good example of how speed is important, customer satisfaction rules, and so forth - this is not.

I don't think it was extremely clear from what she said during her Web 2.0 talk, but my impression is that she meant that searches decreased by 20% (not counting transitioning between pages of search results).

Number of searches seems like a pretty good measure of user satisfaction. It would might be an even better measure if obvious refinements of a search (e.g. "web 2.0 conference" following "web 2.0 summit") were counted as one search.

I do agree that if she meant page views dropped by 20%, that is not obviously a good measure of user satisfaction.

But, I think by "traffic" Marissa meant searches. Google often appears to use searches, not page views, as a measure of traffic, and commonly talks about revenue per search, not revenue per page view.

There's another possible reason, one that is probably more relevant: too much choice. The book 'Paradox of Choice' shows that the more choices you give people, the less likely they are to make ANY decision. Fascinating read with implications all over the place - including for web design.

I would like more detail on this test. A couple of things are bothering me about attributing the reduction in traffic and profit to page load alone:

1) If you display more results per page a user is more likely to find what they are looking for on a single page request. (How often have you gotten to the 3rd page of SERPs without finding at least one link relevant to what you're looking for?) Less page requests needed == less traffic. Someone pointed out above that she may have meant distinct searches. That too can be explained by offering more results. If I don't find what I want on the first page I often refine my search, thus generating another distinct query. If I have 30 results I'm more likely to find what I want on one page without changing my search terms.

2) Google's revenues largely come from adwords which display towards the top of the page. If you increase the page length than a user will end up scrolling down past the ads, thus no profit.

Three major variables changed (speed, and number of results, ratio of ads to search results), not to mention the layout of the page grew to three times the length and she only focusses on the speed as the reason?? I thought employees of Google are supposed to be smarter than that!

My perception may be jilted, but I generally don't make it all the way down to the 10th result. I prefer to click on one of the first 5 or so links. Whatever's on my screen, really. I can usually tell by reading the first couple results whether or not I'm going to find what I'm looking for easily. I do often refine my searches many times, but continue to only look at the first few.

IBM did studies on interactive interfaces in the 80s. The results showed that as the return time neared two seconds, the time the user spent staring at the screen increased. The curve turned up dramatically below two seconds. Below one second, someone had to pry them off the keyboard. And that was with non-rich terminal applications.

You don't have to go far in behavioral science to figure this one out. I wouldn't geek over it too much. Faster is better but it cuts both ways. It is just attention as a product of stimulus/response. Faster torture amplifies the revulsion too. In the balancing act, slower but better search results are still better than faster nonsense.

More choices: that is Shannon's classic signal/noise. Information is that which reduces choice. The more choices, the less information.

No new news in Mayer's comment or the Paradox of Choice. However, it means that any improvement in the means or skills for selecting search terms reduces the choice by increasing the available information. Google should have tried building a few training games into Lively or something like it that sharpens the ability of the user to select the initial search terms.

I seldom need more than ten hits unless I am intentionally widening for 'creative search' where serendipity is used to create something new from the results (a favorite trick for using YouTube when looking for a vid to embed in a blog topic). IOW, I weaken the vector and look for outliers.

I think it is a common mistake to conclude that pages with AJAX or Flash are slower. By slower people must mean that the single page takes longer to load than say, an average page or a similar looking page.

Ajax and Flash displace the "page" metaphor and this is what most critics do not seem to grasp. There is no page when AJAX or Flash are present. So you can't compare a page without such technologies and one with. It would be like comparing apples to oranges. Like comparing a sedan to a bus and concluding that the sedan is faster.

For the comparison to be fair, the AJAX page would have to be compared to the SET of pages that it replaces. Without AJAX or Flash, accessing that information would require loading, sequentially, a succession of pages. The compound load time of those single pages is what Flash and AJAX replace, not the single page. Single pages DO NOT NEED asynchronous technologies, because they are, umm, single.

Because both Flash and AJAX avoid redundant loading by loading upfront and reloading only new data. And because they can effectively manage user experience during load times, something HTML can't do, they can actually minimize the compound load time and, more importantly, the PERCEIVED load time, when properly designed and implemented.

Research from the 80s and the PLATO Project showed that response time really needs to be about .25 second. For something where users expect a bit of work by the computer, stretching that to .4 I guess is OK. But, I would shoot for .25sec.

The math is wrong: You're assuming that search results are the only thing responsible for the page load- Ignoring graphics, other text (links, whatever), tracking code that may be on the page, etc.

Let's say "x" = the time required for a unit of 10 searches, processed and displayed (along with any adsense related to the results)"c" will be the time taken to load everything else on the page not related to searches.From what Marissa said:c + x = .4c + 3x = .9

3c + 3x = 1.2- (c + 3x) = .9 gives us----------------2c = .3c = .15

x + .15 = .4so x (unit for 10 searches) = .25 seconds

Which, now that I"ve done that, could have been figured out just as easily by saying "20 more searches adds .5 seconds, so 10 searches adds .25 seconds, so the rest of the page takes .15 seconds".

Anyway, just an FYI, it's reasonable to assume they didn't screw up the math:P

As others have said, speed certainly matters, but I too am dubious about the Google example.

Of course, I also use Greasemonkey to get an "infinite scrolling" Google search page- as I scroll down, Greasemonkey is fetching the next page of results and displaying them at the bottom of the current set, so that I just never hit the bottom of the page.

How about if they tried new searching algorithm X which takes 10% longer but gives 20% better results - would that lead to "lower satisfaction too", because as someone else mentioned, less searches required does not mean less satisfaction. If you don't get a result on page 1, you'll do one of two things: search for something more specific, so that your original search of "test page" becomes "test page -rabbits +ducks", or go on to page 2. Removing the need for page 2 AND page 3 by raising search result counts to 30 eliminates the need to do the second option, and increases the probability of results that a more specific search would yield showing up.

I have google set to give me 100 results at a time, and by the time I get through the first 10 the next 90 will have loaded.

This test doesn't sound at all representative, there are too many other things that could affect this.

I usually scroll down the first page and if I don't get what I'm looking for I refine my search and try again. With more front page results its more likely I will find it on the front page and less likely i will search again. That's what this experiment would show if i was in it, maybe other people do the same.

It's not about how much time are spent in total, but rather how much are the cognitive process of surfers disrupted between the request and response.

If there's enough of a delay that the users actually NOTICE the delay, there will be A LOT LESS customer satisfaction, than when it's "imperceptibly" small. It's not linear, because there's a lower treshold of about 0.2 seconds before any delay are noticed at all. to 2x the perceptible delay or 4,5x the perceptible delay?

Then there's distractions. If a user is interrupted, there will be enough time to start other cognitive proesses that takes priority, over the original task. Zero opportunities for loss of interest BEFORE the results are presented are also HUGE.

This matters a LOT more than what "makes sense", but that's the way the brain works.

Also, when companies the size of Google/Amazon are willing to share something, they've got both sufficient sample size & methology to be certain. Disregard this if you have better facts.. not because you've got some compulsive need to object.

re: Isaac Rivera's comments: Of course the sedan is faster than the bus! Is it an unfair comparison? No one cares if it is, when the desire is to get there faster, and all other requirements take the back seat.

The rewards of flash and ajax aren't there if the site/page never gets viewed. I know if I navigate to a flash based site, I either click the html link or leave. I don't wait for whatever nonsense the 'designer' has chosen to bless me with.

Why do you make me wait for the end result of your post? Why not just say, "Fast=good, slow=suck. NO MATTER WHAT YOU THINK YOU WANT PEON." If you are going to praise brevity for the sake of expedience at least live by that standard.

What? Are you saying Amazon.com has ever been concerned about how fast their web site loads? That has got to be the most bloated site on the Internet except for eBay. I have to keep Task Manager open and kill at least 6 processes just to give my browser enough CPU to open that resource pig.

By the way, if you are not actually looking for the video but merely confirmation that Marissa actually said this, not only are there other reports on Marissa's 2006 Web 2.0 talk (e.g. [1] [2] [3] [4] [5] [6]), but also Marissa has said similar things in later talks (e.g. her "Scaling Google for the Everyday User" talk at the Seattle Scalability Conference in 2007, see around 13:00 in the video).