This blog is a collection of notes that reflect personal experiences in systems and network administration. "Let all things be done securely."

Thursday, January 20, 2005

What exactly does a search engine do for me?

How many of your friends really understand what a search engine does? I have found that most of the people who I associate with do not understand that a search engine does not search the Internet...what does it search then?...

Even though the popular search engine Google has indexed over 8,000,000,000 pages that you are allowed to search when you use it, you are not searching the entire Internet... You are searching the internet as Google sees it. What does this really mean? Google does not want to keep track of all of the trash and garbage that is on the Internet, they want to keep track of the data that they think people want to know about. Google has developed a very complex algorithm that allows them to make an automated decision about each page that they potentially index based upon some pre-determined and proprietary parameters. One of the greatest challenges to advertisers and marketers is figuring out exactly what a search engine is looking for so that they show up in the search! If you merely create a web page and then search for it on a search engine an hour later, you will not find it there! Maybe an example will help...

Example:
A search engine is similar to a person who is in an occupation where they need to remember a lot of stuff... When you need to know something, you go to this person because they usually have the answer, or something close to what you want. Well, this person has to obtain this information somehow, and has to prioritize what information they want to learn or retain. Google is a very intelligent person capable of retaining a lot of information (it does run on over 100,000 Linux machines) which gives you a very good chance of finding the data that you want. The only problem is, that like most people, Google does not know everything...and probably never will...

If you really had the capacity to search the entire Internet in the time that your favorite search engine returns the results of your search, then the world you live in would have massive amounts of processing power and more bandwidth than you would know what to do with...

Keeping this in mind, this is why it is very good for all of us that there is more than one search engine... We would not want one company to define the internet that we have the capacity to search, just like we would not want one company deciding what we can do with our computer...(unless you use Windows, the you have already handed the keys to Microsoft).