Replicate Google's Panda Questionnaire - Whiteboard&nbspFriday

The author's views are entirely his or her own (excluding the unlikely event of hypnosis) and may not always reflect the views of Moz.

Want to avoid the next Panda Update and improve your websites quality? This week Will Critchlow from Distilled joins Rand to discuss an amazing idea of Will's to help those who are having problem with Panda and others who want to avoid future updates. Feel free to leave your thoughts on his idea and anything you might do to avoid Panda.

Video Transcription

Rand: Howdy, SEOmoz fans. Welcome to a very special edition of Whiteboard Friday. I am joined today by Will Critchlow, founder and Director of Distilled, now in three cities - New York, Seattle, London. My God, 36 or 37 people at Distilled?

Will: That's right. Yeah, it's very exciting.

Rand: Absolutely amazing. Congratulations on all the success.

Will: Thank you.

Rand: Will, despite the success that Distilled is having, there are a lot of people on the Web who have been suffering lately.

Will: It's been painful.

Rand: Yeah. What we're talking about today is this brilliant idea that you came up with, which is essentially to replicate Google's Panda questionnaire, send it out to people, and help them essentially improve your site, make suggestions for management, for content producers, content creators, for people on the Web to improve their sites through this same sort of search signals that Panda's getting.

Will: That's right. I would say actually the core thing of this, what I was trying to do, is persuade management. This isn't necessarily about things that we as Internet marketers don't know. We could just look at the site and tell people this, but that doesn't persuade a boss or a client necessarily. So a big part of this was about persuasion as well.

So, background, I guess, people probably know but Goggle gave this questionnaire to a bunch, I think they used students mainly to assess a bunch of websites, then ran machine learning algorithms over the top of that so that they could algorithmically determine the answer.

Rand: Take a bunch of metrics from maybe user and usage data, from possibly linked data, although it doesn't feel like linked data, but certainly onsite analysis, social signals, whatever they've got. Run these over these pages that had been marked as good or bad, classified in some way by Panda questionnaire takers, and then produce results that would push down the bad ones, push up the good ones, and we have Panda, which changed 12% of search results in the U.S.

Will: Yeah, something like that.

Rand: And possibly more.

Will: And repeatedly now, right? Panda two point whatever and so forth. So, yeah, and of course, we don't know exactly what questions Google asked, but . . .

Rand: Did you try to find out?

Will: Obviously. No luck yet. I'll let you know if I do. But there's a load of hints. In fact, Google themselves have released a lot of these questions.

Rand: That's true. They talked about it in the Wired article.

Will: They did. There have been some that have come out on Search Engine Land I think as well. There have been some that have come out on Twitter. People have referred to different kinds of questions.

Rand: Interesting. So you took these and aggregated them.

Will: Yeah. So I just tried to pull . . . I actually ignored quite a chunk that I found because they were hard to turn into questions that I could phrase well for the kinds of people I knew I was going to be sending this questionnaire to. Maybe I'll write some more about that in the accompanying notes.

Rand: Okay.

Will: I basically ended up with some of these questions that were easy to have yes/no answers for anybody. I could just send it to a URL and say, "Yes or no?"

Rand: Huh, interesting. So, basically, I have a list of page level and domain level questions that I ask my survey takers here. I put this into a survey, and I send people through some sort of system. We'll talk about Mechanical Turk in a second. Then, essentially, they'll grade my pages for me. I can have dozens of people do this, and then I can show it to management and say, "See, people don't think this is high enough quality. This isn't going to get past the Panda filter. You're in jeopardy."

Will: That's right. The first time I actually did this, because I wasn't really sure whether this was going to be persuasive or useful even, so I did it through a questionnaire I got together and sent it to a small number of people and got really high agreement. Out of the 20 people I sent the questionnaire to, for most questions you'd either see complete disagreement, complete disarray, basically people saying don't know, or you'd see 18 out of 20 saying yes or 18 out of 20 saying no.

Rand: Wow.

Will: With those kind of numbers, you don't need to ask 100 people or 1,000 people.

Rand: Right. That's statistically valid.

Will: This is looking like people think this.

Rand: People think this article contains obvious errors.

Will: Right. Exactly. So I felt like straight away that was quite compelling to me. So I just put it into a couple of charts in a deck, took it into the client meeting, and they practically redesigned that "catch me" page in that meeting because the head of marketing and the CEO were like okay, yeah.

Rand: That's fantastic. So let's share with people some of these questions.

Will: And they're simple, right, dead simple.

Rand: So what are the page level ones?

Will: Page level, what I would do is typically find a page of content, a decent, good page of content on the site, and Google may well have done this differently, but all I did was say find a recent, good, well presented, nothing desperately wrong with it versus the rest of the content on the site. So I'm not trying to find a broken page. I'm just trying to say here's a page.

Rand: Give me something average and representative.

Will: Right. So, from SEOmoz, I would pick a recent blog post, for example.

Rand: Okay, great.

Will: Then I would ask these questions. The answers were: yes, no, don't know.

Rand: Gotcha.

Will: That's what I gave people. Would you trust the information presented here?

Rand: Makes tons of sense.

Will: It's straightforward.

Rand: Easy.

Will: Is this article written by an expert? That is deliberately, vaguely worded, I think, because it's not saying are you certain this article's written by an expert? But equally, it doesn't say do you think this article . . . people can interpret that in different ways, but what was interesting was, again, high agreement.

Rand: Wow.

Will: So people would either say yes, I think it is. Or if there's no avatar, there's no name, there's no . . . they're like I don't know.

Rand: I don't know.

Will: And we'd see that a lot.

Rand: Interesting.

Will: Does this article have obvious errors? And I actually haven't found very many things where people say yes to this.

Will: Again, it's open to interpretation. As I understand it, so was Google's. There are some of these that could be very easily detected algorithmically. If you're talking spelling mistakes, obviously, they can catch those. But here, where we're talking about they're going to run machine learning, it could be much broader. It could be formatting mistakes. It could be . . .

Rand: Or this could be used in concert with other questions where they say, boy, it's on the verge and they said obvious errors. It's a bad one.

Will: Exactly.

Rand: Okay.

Will: Does the article provide original content or information? A very similar one. Now, as SEOs, we might interpret this as content, right?

Rand: But a normal survey taker is probably going to think to themselves, are they saying something that no one has said before on this topic?

Will: Yeah, or even just, "Do I get the sense that this has been written for this site rather than just cribbed from somewhere?"

Rand: Right.

Will: And that may just be a gut feel.

Rand: So this is really going to hurt the Mahalos out there who just aggregate information.

Will: You would hope so, yeah. Does this article contain insightful analysis? Again, quite vague, quite open, but quite a lot of agreement on it. Would you consider bookmarking this page? I think this is a fascinating question.

Rand: That's a beautiful one.

Will: Obviously, again, here I was sending these to a random set of people, again which, as I understand it, is very similar to what Google did. They didn't take domain experts.

Rand: Ah, okay.

Will: As I understand it. They took students, so smart people, I guess.

Rand: Right, right.

Will: But if it's a medical site, these weren't doctors. They weren't whatever. I guess some people would answer no to this question because they're just not interested in it.

Rand: Sure.

Will: You send an SEOmoz page to somebody who's just not . . .

Rand: But if no one considers bookmarking a page, not even consider it, that's . . .

Will: Again, I think the consider phrasing is quite useful here, and people did seem to get the gist, because they've answered all of the questions by this point. I would send the whole set to one person as well. They kind of get what we're asking. Are there excessive adverts on this page? I love this question.

Tom actually was one of the guys, he was speculating early on that this was one of the factors. He built a custom search engine, I think, of domains that had been hit by the first Panda update, and then was like, "These guys are all loaded with adverts. Is that maybe a signal?" We believe it is, and this is one of the ones that management just . . . so this was the one where I presented a thing that said 90% of people who see your site trust it. They believe that it's written by experts, it's quality content, but then I showed 75% of people who hit your category pages think there are too many adverts, too much advertising.

Rand: It's a phenomenal way to get someone to buy in when they say, "Hey, our site is just fine. It's not excessive. There's tons of websites on the Internet that do this."

Will: Yeah.

Rand: And you can say, "Let's not argue about opinions."

Will: Yes.

Rand: "Let's look at the data."

Will: Exactly. And finally, would you expect to see this article in print.?

Rand: This is my absolute favorite question, I've got to say, on this list. Just brilliant. I wish everyone would ask that of everything that they put on the Internet.

Will: So you have a chart that you published recently that was the excessive returns from exceptional content.

Rand: Yeah, yeah.

Will: Good content is . . .

Rand: Mediocre at this point in terms of value.

Will: And good is good, but exceptional actually has its exponential. I think that's a question that really gets it.

Rand: What's great about this is that all of the things that Google hates about content farms, all of the things that users hate about not just content farms but content producers who are low quality, who are thin, who aren't adding value, you would never say yes to that.

Will: What magazine is going to go through this effort?

Rand: Forget it. Yeah. But you can also imagine that lots of great pieces, lots of authentic, good blog posts, good visuals, yeah, that could totally be in a magazine.

Will: Absolutely. I should mention that I think there's some caveats in here. You shouldn't just take this blindly and say, "I want to score 8 out of 8 on this." There's no reason to think that a category page should necessarily be capable of appearing in print.

Rand: Or bookmarked where the . . .

Will: Yes, exactly. Understand what you're trying to get out of this, which is data to persuade people with, typically, I think.

Rand: Love it, love it. So, last set of questions here. We've got some at the domain level, just a few.

Will: Which are similar and again, so the process, sometimes I would send people to the home page and ask them these questions. Sometimes I would send them to the same page as here. Sometimes it would be a category page or just kind of a normal page on the site.

Rand: Right, to give them a sense of the site.

Will: Yeah. Obviously, they can browse around. So the instructions for this are answer if you have an immediate impression or if you need to take some time and look around the site.

Rand: Go do that.

Will: Yeah. Would you give this site your credit card details? Obviously, there are some kinds of sites this doesn't apply to, but if you're trying to take payment, then it's kind of important.

Rand: A little bit, a little bit, just a touch.

Will: There's obvious overlaps with all of this, with conversion rate optimization, right? This specific example, "Would you trust medical information from this site," is one that I've seen Google refer to.

Rand: Yeah, I saw that.

Will: They talk about it a lot because I think it's the classic rebuttal to bad content. Would you want bad medical content around you? Yeah, okay. Obviously, again only applies if you're . . .

Rand: You can swap out medical information with whatever type is . . .

Will: Actually, I would just say, "Would you trust information from this site?" And just say, "Would you trust it?"

Rand: If we were using it on moz, we might say, "Would you trust web marketing information? Would you trust SEO information? Would you trust analytics information?"

Will: Are these guys domain experts in your opinion? This is almost the same thing. Would you recognize this site as an authority? This again has so much in it, because if you send somebody to Nike.com, no matter what the website is, they're probably going to say yes because of the brand.

Rand: Right.

Will: If you send somebody to a website they've never heard of, a lot of this comes down to design.

Rand: Yes. Well, I think this one comes down to . . .

Will: I think an awful lot of it does.

Rand: A lot of this comes down to design, and authority is really branding familiarity. Have I heard of this site? Does it seem legitimate? So I might get to a great blog like StuntDouble.com, and I might think to myself, I'm not very familiar with the world of web marketing. I haven't heard of StuntDouble, so I don't recognize him as an authority, but yeah, I would probably trust SEO information from this site. It looks good, seems authentic, the provider's decent.

Will: Yeah.

Rand: So there's kind of that balance.

Will: Again, it's very hard to know what people are thinking when they're answering these questions, but the degree of agreement is . . .

Rand: Is where you get something. So let's talk about Mechanical Turk, just to end this up. You take these questions and put them through a process using Mechanical Turk.

Will: So I actually used something called SmartSheet.com, which is essentially a little bit like Google Doc spreadsheets. It's very similar to Google Doc spreadsheets, but it has an interface with Mechanical Turk. So you can just literally put the column headings as the questions. Then, each row you have the page that you want somebody to go to, the input, if you like.

Rand: The URL field.

Will: So SEOmoz.org/blog/whatever, and then you select how many rows you want, click submit to Mechanical Turk, and it creates a task on Mechanical Turk for each row independently.

Rand: Wow. So it's just easy as pie.

Will: Yeah, it's dead simple. This whole thing, putting together the questionnaire and gathering it the first time, took me 20 minutes.

Rand: Wow.

Will: I paid $0.50 an answer, which is probably slightly more than I would have had to, but I wanted answers quickly. I said, "I need them returned in an hour," and I said, "I want you to maybe have a quick look around the website, not just gut feel. Have a quick look around." I did it for 20, got it back in an hour, cost me 10 bucks.

Rand: My God, this is the most dirt cheap form of market research for improving your website that I can think of.

Will: It's simple but it's effective.

Rand: It's amazing, absolutely amazing. Wow. I hope lots of people adopt this philosophy. I hope, Will, you'll jump into the Q&A if people have questions about this process.

Will: I will. I will post some extra information, yeah, definitely.

Rand: Excellent. And thank you so much for joining us.

Will: Anytime.

Rand: And thanks to all of you. We'll see you again next week for another edition of Whiteboard Friday. Take care.

I have a passion for both learning and helping people. I'm often found up late at night coding or designing something for my clients or one of my many secret projects. I'm a huge fan of tracking and trying to figure out what Google is up too! Feel free to follow me on Twitter: @caseyhen

This is starting to sound like the marketing I was taught in school. Using surveys and feedback to increase quality :-) Amazing how Panda is getting SEO's to stop thinking "gaming" and start thinking about a quality user experience.... + links. ;-)

To take it one step further I think targeting actual site visitors with the survey instead of mechanical Turk would yield better answers. Being panda proof is the lowest acceptable level now. Why just shoot for that when "crazy awesome" is available?

I really enjoyed this whiteboard Friday as the Panda update has been a very common topic of discussion with many of my clients, there is SO much mis-information out there about it.. The fact is, that Panda is just what it says, a "farmer's update", Google is targeting websites that are there to "farm" viewers for monetary reasons, not geniune ones.

Internet users are getting more and more intelligent in regard to online advertising, both honest, malicious, and just plain gimmicky. They are actually starting to READ and EVALUATE the content and quality of a website before clicking away. Although, many viewers may value design as an equal to content, content has become a stronger focus as the average webviewer has evolved. Design is just a backbone to support the delivery of content and give the foundation for gaining trust with your traffic, and eventually making conversions.

Your quote, " And finally, would you expect to see this article in print.?", is so accurate.

The true future of internet marketing, and success online, is to build quality, unique AND USEFUL content that sells through its words, concepts, and helpfulness to it's readers. Not through spammy banners, junk mail, and tricky CPA ads. You can only fool the masses online for so long and Google knows that over the past few years, that the public has become more scrutinizing to its search result quality. Panda is a direct result to keep up with the masses intelligence and understanding of our global webspace.

Google must clean-up the webspace and evolve its search for higher quality and keep their trust, or they have a limited future themselves.

Great video, alot of people have been doing it tough online especially the small business clients who have been hit hard by panda who have been working with articles which are not the best.

But the thing is I usually deal with the larger end of the market. In the cases where we need to fix problems which have been caused by Panda usually a slides deck is the best way of doing things, yet then you have to get web dev teams to move quickly which some times can be a hard task. Even if you have senior management and directors involved and on board.

Thanks for the interesting idea. We tried this out but found out, that over 90% of people answering are from India. So keep this in mind, if your questions imply, that there are answered from people in the US or in the UK...

You know.. after reading this whiteboard friday, it reminds me of my collage years, We learn one thing as a software programmer, we ALWAYS deemed the software we created as User Friendly, but if we want to find the REAL answer about "how user friendly is my software", than you have to take surveys...

And indeed, this posts reminded me that the very same technique we employ to determine if our software is user friendly can actually be used to determine if our website has a great content...

I think that is a great way to measure the quality of an online article. So often the way we write online pales in comparison to the way we write offline. Some say that's because online readers approach reading differently. But you can still create great content that is online reader friendly.

The best thing of using this method in order to verify your website is it's simplicity.

Just a question... From what I understood, this technique is perfect especially for sites that has to be launched.
In the case of websites that are constantly adding new content, don't you think that a good alternative to the mechanical Turk "quality" evaluators could be forms like the ones you can create with kiss insights, maybe adding a symbolical incentive in order to have the most people filling them? This solution could possibly be good also to have a really big amount of datas and so better verify if your site in responding well to the panda factors.

Excellent idea, and so simple to execute, thanks for sharing this with us Will! Although, I hate to say that certain members of my team were somewhat distracted from the awesomeness by Will's similarity to politician David Milliband: http://bit.ly/WillandDavidLongLostTwins - sad but true.

Very interesting indeed. Especially the reference to "medical" content. Having launched 6 month ago a new medical site (migraine.com), the panda update have generated a significant increase in our traffic patterns (>25%) and the fact that there are mostly poor quality content in medical field (as opposed to expert driven, referenced, print quality articles etc.)

Am I surprised that there are no mention of links in your questionnaire? Is there any specific reason why?

Thank you for this WBF. I wrote a post on my personnal blog (in french) base on Matt Cutts interview and Google post, to give some advices to webmasters in order to help them prepapre Google Panda and i got good feedbacks...I guess you must make another WBF to give a short checklist to help SEO experts who hasn't face Panda yet, to avoid it :)

What I want to know is this: when you got the feedback, did you then apply it to your site?

If you did, did you come out of Panda?

At the moment I feel this WBF only presented half the story. There are no results mentioned at all. We need to know, did it work? Otherwise this is just speculation about what Panda is about (amongst a gazillion theories out there).

Nice information, after Amit Singhals Panda tips...but another thing is people use to see how many people shared/Likes that article to Facebook,Twitter for less branded or less authoritative website before endorse his user trust factor on that writing..

Pudding Creative does a lot of work with the aesthetics of websites. We make sure that high levels of functionality are matched with equally attractive design and engaging text. This information is great for what we do – and for making it better.

I did a little bit with Smartsheet.com --- but it seems like there's no way to really target segment. OR Decrease the number of spammers. Makes me wonder if smartsheet really is the best idea to get quality results.

Don't get me wrong - I liked the questions, BUT is smartsheet.com really the best for this type of job -- especially when we're trying to understand what Panda would analyze.

Unfortunately, when I ask myself some of the questions you guys tackled in tthe video and then look at some of the sites I do SEO for... Well, at least I can say I have plenty of things to do for a very long time ;-)

Checking the UI for any page is always stupidly important and should be done as a constant so this is good to add as another layer of testing for site user interface as well as helping you to look at it from a different angle and push towards an SEO friendlier site.

Thanks for this Whiteboard friday great subject to push us SEO lot towards with the latest Panda update.

It may be dirt cheap, but that doesn't make it any less of a gold nugget, not just to get the clients and customers to listen to you, but also for webmasters and bloggers to survey their own site and get to work on their own content...

from what I have learnt from this is that Google is looking for sites that build trust and have crediable content. This makes total sense for larger companies but when your dealing with a small companies then this is going to go straight over their head. I really think the questions are pretty much the same...

I couldn't help thinking all the way through this I could put together an agency managing this kind of work for SEOs who want something a bit more than just a few fields filled out; if you wanted more detailed user interface reviews, or 'Panda-esque' analysis why not get students to sit down for 10-15 minutes and go through a website in detail. Record their facial expressions - what would make good content for slidedecks etc.

"get students sit down for 10-15 minutes and go through a websites in detail" seems good but not a good idea because that depends on the kind of business. According to me customers or visitors of a website have the best position to judge the quality of a website.

One of my client, use this technic (send questions to non-buyers) on a regular basis to find ideas of content improvents and its works: better bounce rate and time to website, more return visitors, much sales,etc...

However, as a teacher i've conducted some surveys with students and results were clear : if you need a general appreciation of the web (facebook-Google, what is the best website?) students are the good target; but if you need more specific ideas, or assess your Ecommerce website content for example, therefore you must find a way to ask questions directly to your buyers and non-buyers visitors

That your content offers something useful for the viewers you are targeting. It is well-written and intelligently designed so that the concepts and ideas are easily translated to your audience. In SEO terms, I would say that in a competitive niche/market, EXCEPTIONALLY GOOD content comes in well over 1,000 words per page + media. You see here on SEOmoz they make it a point to keep their content well in the 1,000 word+ range, always including numerous images and videos as well.. All of which are UNIQUE.

I'd say 90+% of internet marketers are not creating content in that range, that is why we have seen many large content, authority sites pushing the little guys out of the SERPs.. Often times, those admins who atleast write their own or outsource unique content, rarely get their pages above 400-600 words.. Even if you've got more linking power, you'll lose to much higher quality and quantity of content webpages. Thus why we see a lot of IMrs whining about the panda update, "they killed my crappy 10 page ad-sense site, but I have 10,000 backlinks!!".. Hmm.

Yes very much true, but arround 40% of bloggers copied content from news sites, and they are in rank becuase Google didn't hurt news site ranking becuase news are copied then how will you justify that this content is unique and exceptional if we talk for news sites: like, if someone killed every news sites write this man has been killed nothing else right...and one of my competitors do this with same keyword domain, and you know what he is always on top in SERP never hurt in ranking and the reason is he is copying content from news sites and update 10 posts per day, and remains at top no backlink nothing else simply copying the content with same domain name , it is so much frustrating for me.

i can't understand, why most of the time member didn't reply answer but always make my question tumbs down, its realy not a good thing, share your thought first. or please SEOmoz, add names who are making us thumbs up or down.

Man, what a list! This checklist should be given to everyone that wants to make a website. I was just bitching on twitter earlier (@Austadpro :) about all the junk pages online. I think Google needs to keep working on this. Maybe if they worked with other web companies they can come out with a standardized list like this that webmasters should follow. Great Job Will!

That's a fascinating analysis of the Panda effect. I understand why Google are doing this and why they should be making it hard for us SEO types to succeed. I haven't got a problem with that at all - other than current difficulties in ranking ;-( The thing that I do get annoyed at the almighty, high and mighty G about though: is the inconsistency and opaqueness with which they apply their often draconian slaps and account lockouts. What do other people feel about that side of the story? I'm speaking as one who hasn't really been messed about myself. I have seen perfectly good websites by honest and diligent friends eclipsed by absolute garbage. Nowhere more so than in the local arena!

Thanks Will, and all that information for free! it is obvious what should be happening but seeing all the questions as a checklist is really helpful, Especially on the blogside of things and after thinking things through it was clear to me using that check lilst some of my blog posts were vastly superior to some of the others or Visa versa

As much as I love whiteboard friday (and I DO love whiteboard friday), I have a sneaking feeling this was filmed on a thursday (or earlier) and certainly was posted on thursday. I can't believe you'd do this to us!

That is sure... And it is normal to record stuff before the actual day of broadcasting (it would be a live WBF the other way). Anyway... The world is not USA centric, and in Europe it was actually Friday when this WBF got published :)

I just submitted www.ravenshoegroup.com to a random and you are bang on about the cheapest market research ever! We had just sent this to 50 random people and paid $0.50 an answer with a reponse rate of 46%! for a whapping $126.50 we recieved some valuable feedback from the questionnaire and after analysis some pretty significant upgrades to our content.Offering this research to my clients is the next step!

Great article, i think google web master posted a lot of theses questions before. Some of these questions are kind of vague. You could compare this to the mortgage industry crackdown on bad brokers pedaling subprime loans. It will hurt some people but makes the rest improve and go back thinking more. thanks