Google Finance, Govt, Policy and Business Issues Forum

Fact: Google has developed the technology to vary the frequency and depth of crawling for specific sites, depending of any parameters of their choice. Evidence: They already do vary their crawling intensity depending on the average PageRank of existing sites, as confirmed by several people in other threads at WebmasterWorld.

Fact: Accepting payment for more frequent crawling doesn't necessarily harm the ability of an engine to maintain the quality of their index. Evidence: During the last two months, and nonwithstanding PT, Inktomi has taken over the lead from Fast in terms of adding fresh and unpaid listings to their database. And if you don't like the relevance of the Inktomi SERPs, then dropping PT probably wouldn't change anything about that either.

Fact: Only a very small number of specialized (and big) sites would really benefit from daily crawls by Google (other than for SEO experiments). Evidence: I haven't heard any complaints here about the fact that Google has recently switched to one of the fastest crawling cycles in the industry for free.

Fact: The decision of the Google founders to put the business matters into the hands of business people was a good and necessary one. Evidence: Many of those who didn't can be found here [webmasterworld.com].

Fact: An unnamed writer at ZDnet has jumped to conclusions and designed a new business plan for Google, solely based on fact number one above. Evidence: See link at the top of this thread.

Ok, so you will certainly find counterexamples to most of my "evidence". But all things considered, and this time without any provovative intent, I really don't understand what the excitement of this thread is all about.

And no, I'm not working for Google, as some stickymail suggested. I am a part time webmaster, besides running my own software business. But I am also an internet veteran (before the www), and I have observed the big engines and directories carefully from the very start of their existence. More often than not, I find the discussions on WebmasterWorld helpful and well informed, but in this case I simply fail to see much rational reasoning that I could follow.

Of course itīs just fine for Google to deliver fresh results. Now who needs results to be fresh? Media companies and companies selling media like CDs, DVD, books, videos are the first to come to mind. Lets imagine they have to pay for daily indexing, which is for those big sites a lot of data to compute, how would those companies react when being buried somewhere on the serps? I do not mean to doubt the integrity of Googleīs - and for that. Fastīs - ranking. (on the other hand...) But imagine what a media giant could do to Google! So if a change in algo was planned wouldnīt it be wiser for Google to kind of put that media giant on the trail before itīs effected? Or that giant has accidentally triggered some spamfilter, well would not a slight adjustment come in handy? Taken into account plans for going public with a massive pressure from stockholders for profitability Iīd like to come back to what rcjordan has said before: this is a massive slippery slope issue.

And no, I'm not working for Google, as some stickymail suggested.I'll just let that stand on it's own. Being primarily a programmer you may not know how the US corporate machine works, even if you are intimate with the technology. As pay-for-crawl is injected into google's db there will be an incentive to persuade sites into their revenue stream. The easiest way to create incentive is to corrupt the free listings in some way. The commodification of database real estate will undoubtably taint results, as inktomi has proven.

>>More often than not, I find the discussions on WebmasterWorld helpful and well informed, but in this case I simply fail to see much rational reasoning that I could follow.

The rational reasoning behind this thread comes from past experience. Every single search engine/directory that now has some type of paid placement/inclusion program all publicly denounced the idea of ever charging fees. The most recent example can be found in here. [webmasterworld.com]

Another interesting point is that all of the spidering engines that now offer paid inclusion seemed to dramatically increase the frequency of their crawls prior to the introduction of their paid programs.

What is different in Google's case is the fact that so many people seem to not want to even consider the possibility that Google's considering introducing some type of new, for-a-fee service.

I think that comes from the fact that most webmasters (myself included)really want Google to succeed. If they have actually "built a better mouse trap" and they are able to reach profibility (which according to recent reports, they have) without traveling down the paid inclusion road, then others will follow. And having more free-crawling engines (like the old days) that actually produce traffic is something we would all like to see.

However, the current competitive landscape is one that makes the idea of sticking to their current model not an option. As Google's marketshare continues to increase, their portal partners will realize that they have gone from being a backend provider to being a competitor. As their partners continue to struggle with their own financial problems, they may very well come to the realization that paying alot of money for secondary search results provided by a company that is poised to start pulling search traffic away from their sites, isn't the smartest business decision.

Both Ink and Fast are in a position to easily replace Google as the portal partner of choice because they not only have equally large databases, they also have the ability to offer partners a revenue sharing program similar to Looksmart's. In these tough financial times, moving from a partner that cost you money, to one that makes you money seems like a no-brainer. And I'm sure that even someone who helped drive Novell into the ground can see that.

When you add in the pressures they will face from stockholders once they go public, (a minor inconvenience they have not yet had to deal with) I think it becomes pretty clear that they will launch some type of program similar to Ink's Index Connect or AV's Trusted Feed program.

I don't see them launching a PFI program for the average webmaster, but grabbing their share of the cash that companies like Amazon and Ebay are dumping into inclusion programs, is something they must do.

<devils advocate> The commercial pressures are primarily driving the paid schemes. The other aspect is, simply, the opportunity for a business to make money. If the paid schemes fail to achieve a profit they will either go to the wall or re-invent themselves. An unpaid scheme will only work with income from elsewhere. Traditionally, paid for by advertising revenues, unpaid engines will need to be lean and mean.

With advertising revenues declining, where else do we see revenues? </devils advocate>

The ZD story does not appear to be based upon fact - perhaps heresay. It may be a plant to generate exactly this kind of speculation.

Well, there's a reason why I never really cared much for Inktomi the company. As a business, Inktomi really has done everything to lose my personal trust. At the same time, I don't have any problems with them from a SEO point of view, because a reasonable amount of link popularity works perfectly well with them in my little corner of the web.

Google is a completely different story in my book. As far as I can recognize, they have always done the Right Thing, even if it wasn't always immediately obvious to everybody (anyone still complaining about the deja.com take over?). Raising hell over them now, just based on obviously unfounded speculations, really doesn't seem like the rational thing to do. Google has found refreshingly different solutions than the rest for both their search services and for advertisements on their site in the past. Why not assume that, if they should go some kind of "pay for whatever" route, that they will apply their creativity again and make it a pleasant experience for everybody?

I don't think that the equation "Inktomi has done it in a bad way, so Google can't do better" yields any useful results, but of course, only the future will tell.

They could make *more* money by selling subscriptions to their database. Like the goto tool, only you have a small fee to use it. If it were kept to an amount that most people could afford, everyone would win.

If webmasters know what people search for, they can gear their websites to cater to that.

Searchers win because they are more likely to find information they need.

Webmasters win because they know what searchers want to find.

Search engines win because they make money on the deal.

Win win win.

My favorite non-google engine: Northern Light. [northernlight.com] I have just started using it recently, and for looking up research topics, it can't be beat. I especially like the special editions. [special.northernlight.com]

But who's mouth? It was a well orchestrated campaign backed up but the power branding of the Google page cache. I looked long and far for those promoting Google and they all seemed to have the same lingo - eg: company by lines written by marketing officials. From forums to newsletters to news stories. I think the "word of mouth" campaign was a a few pc's at the Googleplex marketing desk and some well placed "advertising" dollars (we used to call it payola).

>As far as I can recognize, they have always done the Right Thing

And your feelings on the Google "cache"? (term used loosely as it's not a "cache", since true cached pages are not injected with branding advertisements)

I feel at worst that it is illegal. I feel it is a flagrant breaking of all applicable United States and International copyright laws. I do not believe Google has but a thread of a legal hope if they are challenged on copyright in court. The only legitimate claim they could put forth for protection might be the ISP exclusion rule of the DMCA. Other than that, there is no gray area and no justification for it.

At best, it is unethical. In the past, Google employees have made statements to the effect, that they have more "PHD's per square foot" than any other internet operation. Every academic person I've ever known has an almost biological disgust at plagiarism. By placing their advertisement on the cache page and consequently in the url address bar, that is exactly what Google is doing: plagiarizing (or as we call it: Page Jacking) 1.3billion internet pages.

They have never once -- not a SINGLE time -- explained or justified it in print or on their website. I feel they don't do it, because they know it could be used in potential legal action against them. Their lack of public discussion about the page cache, taints EVERYTHING Google has to say and do.

We can't count on Google for information about Google. Their own employees on numerous occasions have made contradictory remarks in the exactly same week. So when an independent and mostly legitimate outlet such as ZDNET puts forth a story about them, you bet we are going to give it serious consideration. Especially given the history of ZDNET's stellar (somewhat suspect) rankings in Google, and their unwavering support of them. Given that history, when ZDNET talks about Google, I'm going to listen with an extra measure of concentration.

As someone else mentioned, just a couple of months ago, we spotted an engine headed for PFP and we trusted them. We were let down. It's that old "fool us once..." adage thing again.

Lastly, many of the people in these forums and their clients have a great deal riding on Google remaining the way it is now. For right or wrong, they've been forced to bet the bank on Google as the only remaining source of quality free traffic. That's both for commercial sites and personal sites. I'm sure you noticed that their has been just a slight "down turn" in the internet economy the last year. Google is the difference between many of these sites making it or being gone.

A wake up call of this nature has been far too long over due. Those that are now caught in the Google trap are looking for alternatives.

Along Comes WiseNut. They represent a new avenue for both promotion and for searching. I feel they have been relatively overlooked.

They have some of the most tech savvy people a startup search engine has every put together. Read their "team" pages - it is a who's who of data mining, internet management, and marketing. It's an incredible team of experienced people they've put together.

They are backed by people such as SamSung and AmBex (who is owned by Chong-Moon Lee the founder and top shareholder of Diamond Multimedia). [wisenut.com...]

It's Huge: They have the largest database of any search engine on the net now and are producing very accurate results. I feel when you get into the 3 and 4 word combo's, they are more accurate than Google.

All that says to me is that they are a player and will be around for some time to come. I think they are the best startup search engine to hit the net since Northern Light.

I kinda look at Google like my girlfriend (ok not smart comments here).

If they did what some are suggesting - I would be disappointed, hurt, and betrayed.

I agree with easyd, ciml, and bird on this one - unless there is more to the article I missed - it is CRAP. Sorry, there is no other word for it.

Someone on here posted that they heard an NPR radio interview (I think that is what it was) with Page and that he wanted to do more frequest crawling.

This is not ANY EVIDENCE that they are going to payper anything. I know that everyone on here is not fans of Google, but they ARE a better and smarter company out there than anyone else.

Fact:

Google has never blamed their troubles on spammers, while AV and Inktomi always do.

AV Has a freaking ransom note you have to use to get your page crawled - what for??? They don't seem to bother to index you.

Google says "submit all you want"

Why, because they are a search engine, the type of search engine that was what search engine meant to be - when men were men and women were women.

When search engines actually bothered to crawl the web - and didn't hit someones robots.txt file 4,100 times without visiting their site.

Search engines that bothered to give relevant results to consumers and took responsibilty to make sure their SE could deliver - with minimum human intervention.

Search Engines that did not try to make a quick buck, but tried to help advertisers and consumers alike. With the clearly more effective text ads that smarter advertisers know are effective than the standard 468x60 banner.

A search engine that embodies the spirit of the Internet. The only search engine with the cajones to put an "I'm feeling" lucky button on their page - confident in their belief that they can deliver.

I am a sceptical, but I have faith in one Internet company and that company's name is:

[big]GOOGLE[/big]

I'll put that faith against some ZD guy's conspiracy opinion anyday. Perhaps Google is just trying to be better.

Most of the other SEs that have gone this route have run into trouble BEFORE they did so.

I am not sure the technical difficulties of running a search engine, but only fast/ATW seems able to compete - and they don't have nearly the popularity of Google.

Time & Gateway Search Wisenut: Wed May 30 Teoma: Dec 2 10:26:54 GMT 2000 Altavista: Jun 30 16:24:38 GMT 2001 Hotbot: Tue Jul 31 19:57:40 GMT 2001 (But they also have 2000 results on the same page and one from 1994!?! - I didn't even know they existed then) Excite: (couldn't get it as text was cutoff) - next result: 2001/05/21:08:24:18

Google: Aug 6 05:33:52 GMT 2001

Teoma and Wisenut look promising, but still have a lot of work to do to match up to Google - and not just # of pages.

I wasn't sure if you were being sarcastic Brett - or if you believe they will go this way too.

There simply is no need to do so. I think they can so fine off their current business model. It would ruin their reputation if they went to PPC or PPI. I don't think they are that greedy or that short sighted.

"Inktomi, AltaVista and Fast Search are just a few companies allowing marketers to pay for prominence in search results -- a trend that came into vogue with the success of for-fee engine GoTo.com."

Oh, and I don't know if I would call blowing through a half billion dollars last year a success. My idea of a success is a PROFIT.

While "Observers have wondered whether Google's business model that can survive, especially given the downturn in Internet advertising. Schmidt, however, insists the company has been profitable for the last two quarters, though he declines to disclose numbers." [thestandard.com...]

PayPer Click is no magic bullet - they have to hire editors and all the associated crap with that, and PayPer indexing is shooting their customers in the foot.

I guess my opinion is that PPC is definitely no assurance that you will make money - they would need much more money to start such a venture.

And, although much more feasible, I just don't think PPI would work out for them. What that basically means is that they won't index sites that don't pay them.

I know that companies don't like to say it, but that is what it boils down to. You have to pay protection money.

I just can't see Google doing it. People use Google because it is a good search engine, not because they are tricked into it by their ISP or it is the default on their browser.

Maybe I am naive and yes I know I am biased towards Google, but I think for good reason.

According to Schmidt, they are already making a profit. I understand what you are saying about the IPO thing, but that doesn't mean it is true in this case.

So far, the only proof is that they have increased indexing cycles. They should be doing this anyway. This does not mean that they are going this way. They seem to be only doing it to high PR sites. I get crawled 4 times a day now.

Assuming this is evidence of their move - shouldn't they be crawling sites at the direction of some other factor than PageRank - like test sites that are "paying" them? And why four times a day? Wouldn't it make sense to index more sites once a day - than 1/4 as many four times a day? Are they going to offer a 4 time a day crawling service?

I just don't think it adds up. I don't think they would make more money. Maybe $1 million in the short term. I know I am just pulling that out of my butt, but that would require a profit of $100 each from 10,000 companies.

I guess we will see. I am getting in three Google shirts this week and am willing to place one a side for anyone that wants to wager differently :)

That's just to build it, that's not to service it. There are huge "care and feeding" bills with an se.

Go to your bank and withdraw $10,000 in hundred dollar bills. Set them on the back of your toilet. Each morning grab one bill and flush. When it no longer bothers you - you are ready to start a major search engine.