I often see tales of woe posted on messageboards by webmasters who
invested everything they had into their websites, then worked for months
or even years with little to show for it. Why do some succeed when
others fail?

The success of one's website
depends on many factors but one of the most significant is how well it
ranks in the search engines for the keywords relevant to the content of
one's site. If one's site is in the top ten, so that it appears on the
first page of search results, the search engines refer a great deal of
free traffic. If one's site only appears after clicking through page
after page of listings, there will be few search engine referrals and
one must depend on expensive advertising to draw traffic to one's site.

Search engine optimization, or SEO is the practice of improving
the design and content of websites so draw more search engine traffic.
As we shall see, there are ethical and unethical ways to accomplish it.
I advise you not to hire anyone who does not pratice ethical SEO. You
should be able to make the most important optimizations to your website
yourself after reading this article, and if you are a web designer, reading
this will enable you to offewr ethical SEO to your clients. If you plan
to hire a consultant to optimize your website, reading this article
will help you understand what they will do for you, and to figure out
if they are really Black Hat SEOs whose work might well pay off in the
short term, but yield a disastrous collapse in your traffic once the
search engines catch onto their game.

Contents

The business of search engine optimization is infested with sharks:
scammers, swindlers, spammers and fraud artists. They are the Bad Guys
of the Internet, and so like the Bad Guys in the old cowboy movies, we
say they wear Black Hats. You could tell the good guys in the cowboy movies
because they wore
White Hats.

The practice of White Hat Search Engine Optimization, also known as
Ethical SEO, aims to make
it easier for the search engines to tell when your site's content is
relevant to a query. In most cases we do this by making a site genuinely
more useful to its human visitors, so that they will recommend it to
others by linking it from their own websites. The most important White
Hat technique is simply to have a website worth recommending. That is
not easy by any means, and certainly can require a great deal of work, but
it is the honest way, and the only way likely to yield lasting results.

But for every White Hat technique there is a Black Hat technique for
simulating it. While I work hard to write articles that help one understand
various topics that often turn out to rank well for the relevant keywords,
the Black Hats write software to generate thousands of grammatically correct
yet meaningless articles filled with keywords carefully chosen to draw
search engine traffic and yield maximum ad revenue. While I depend on others
to recommend my writing by linking my articles, the Black Hats operate link
farms, thousands of pages all linking to the single site they wish to promote.

Black Hats of all sorts are known as "hackers", but that is a name they appropriated
from those who originally wore the badge with honesty and pride, those who
work hard and cleverly to develop computer software and hardware. I have been
proud to call myself a hacker - the White Hat kind - for many years, so I
prefer to call the dishonest ones Black Hat hackers. The Black Hat SEOs have
the specialty of hacking the search engines to make money dishonestly. White
Hat SEOs by contrast work to make it easier for the search engines to do a better job.

While you may not care to follow the straight and narrow path,
I assert it is in your best interest to do so. There is nothing illegal, not
yet anyway, about being a Black Hat SEO, but all of the search engines employ smart
people who work hard to defeat them. They have to, because the success of a
search engine company depends critically on delivering search results that are
as relevant as possible to each query and useful and interesting to the searcher.
Whenever a Black Hat succeeds by capturing a search engine's traffic, their
visitors soon figure out they have been had and instead try their search on
a different search engine.

It is in fact quite possible to make a great deal of quick, easy
money by spamming the search engines. But I assume that if you are
reading this article at all that you have a website whose long-term
success you are committed to. Ethical SEO yields the stablest, most
secure long term results:

From time to time each search engine improves
their search algorithms in ways that tend to defeat search engine
spammers and promote legitimate websites. Even if you are able
to escape notice by flying under the radar, an algorithm change can,
and almost certainly eventually will bring your site's traffic
crashing down.

If you succeed with your unethical website optimization well
enough to become a high-traffic site, you could fare even worse: you
might get caught, as WordPress
did when they sold hidden links of their high PageRank homepage to help
advertising publishers capture search engine traffic, and as a result
was
removed from Google's index. As GoogleGuy said:

There definitely appear to be hidden links on the root page of
wordpress.org using CSS, e.g. "text-indent: -9000px; overflow: hidden".
That's clearly against our
quality guidelines...

Google's guidelines are quite clear on things like hidden text
and hidden links to duplicate content. People should have a skeptical
reaction when someone comes trying to buy links to spammy/duplicate
pages, esp. if they want control of a subdomain or a subdirectory on
your own site--linking to content like that can trigger effects to
a whole site's reputation...

There is no ticket to easy street. Success on the web, lasting
success, takes long hard work. A search for "search engine optimization" at
any of the search engines yields page after page of both result listings
and advertisements claiming top ten placement and quick results. I don't
even have to look to know they are frauds. Let me make this
absolutely clear:

Results are never guaranteed.

Results are never quick.

Don't quit your day job. If you work both smart and
hard on your new site, you could be earning a living from it in a year.
But it will likely take months before you get many search engine
referrals. Once a site is well-established in the search engines,
referrals to new content can show up in less than a week. But it
takes time before the search engines learn your site is worth treating
well, time for your visitors to learn about your
site and for some to link it from their sites, and time for the search
engines to learn that your site is so highly regarded by others.

How do search engines decide where to present each page in their results?

A search engine's objective is to present each result in the order of its
relevancy to the search query. How to do so in an automated way, for every
page on the Web and for every possible combination of keywords in all the
languages people speak is a very deep and difficult
problem. Search engines keep most of their methods - their
algorithms - secret, in part to make life more difficult
for the Black Hat SEOs and in part to maintain their competitive advantage over
competing search engines.

One algorithm is known publicly, the PageRank algorithm
invented by Google founders Larry Page and Sergey Brin when they were
graduate students in Computer Science at Stanford University. Its details
were published when they were granted U.S. Patent
number 6,285,999 for PageRank's invention. The word "PageRank" is a trademark of
Google Inc.

Unfortunately, public knowledge of the PageRank algorithm allows anyone
to manipulate the PageRank of their site. Thus the PageRank Google
calculates for real search results, while based on the patented algorithm, is much more
complicated than I describe here, mostly to defeat the efforts of the
Black Hat SEOs.

PageRank measures the probability
that one would end up on a given page when clicking links randomly while
surfing. If one ends up at a dead end, that is, a page with no links, a new
page is selected randomly.

Google will tell you the PageRank of any page on the web if you install
one of the browser plugins listed below, . The Toolbar PageRank is an
integer from zero to ten, but the PageRank used calculated by Google is
a floating point number that is scaled and rounded for the Toolbar.

If two pages have otherwise equal PageRank, Google will present the one
with higher PageRank first in its search results. Pages with higher
PageRank are thus considered more important or reliable by Google.

The success
of the PageRank algorithm quickly propelled Google to the dominant position
among the competing search engines. This combined with the fact that
PageRank is easily measured quickly transformed the practice of
search engine optimization into a never-ending quest for higher PageRank.
While PageRank is important, and will help you to improve your
PageRank, it is by no means the only factor in your success, as we
shall see.

PageRank can be understood as a democratic process: each page on the
web has one vote, and distributes that vote among the other pages it
links to. The votes of pages that themselves have higher PageRank are
given greater weight in the results.

If more pages link to one of
your pages, its PageRank will increase. But it is important to understand
that it is the PageRank of the page on which your link falls that counts,
not the PageRank of the homepage of the site that gives you a link. I am often spammed with
link exchange requests from sites that brag of their high PageRank, only
to find that their link pages are buried, and have low PageRank. There is
still value to swapping links, but not as much as the spammers claim.

A page that has no inbound links is given a small, but nonzero PageRank.
The ToolBar PageRank will be reported as zero, but the value Google uses
is a small fraction.

The PageRank contributed by a given page to those it links to is
equal to its own PageRank divided by the total number of links it has.
Thus if I have a PageRank five page that links to one of your pages,
but my page has four other links on it, it will increase your page's
PageRank by just one.

The PageRank reported by the Google Toolbar and the other plugins
listed below is an integer scaled to fall between zero and ten. It is
thought to be logarithmic, so that it is much harder to get to the next
step than it was to advance from the previous one. No one outside of
Google is sure what the base of the logarithm is, but let's suppose it is 10. A real
PageRank of 1 would then have a Toolbar PageRank of 0, while PageRank 10 gives
a Toolbar PageRank of 1. To achieve a Toolbar Pagerank of 10 requires
a PageRank of 10,000,000,000.

From here on, when I refer to "PageRank", I mean the Toolbar PageRank.
It is often abbreviated as "PR".

I worked for seven years to build my previous company's
website with the result that its homepage rose to a PageRank of 6 and some of
my other pages were PageRank 5. After focussing my efforts on
GoingWare's Bag of Programming Tricks for several
months, it rose from 5 to 6 and a few of my other pages rose 4 to 5.
I am working now to get all my articles to PR 5. Some
of them are still 4, and a few are 3.

However, I didn't have much of a clue about the webmastering
business for most of that time. If I had it all to do over again
with what I know now, and worked at it full-time, I expect I could
achieve the same results in a year. That's why I said earlier that if
you worked both smart and hard, you could earn your living from your
website in a year.

SearchStatus also reports the Alexa
Traffic Rank for a site. Alexa's rank is reported for the whole domain a page
falls in, rather than for a single page. Internet Explorer
users can install the Alexa Toolbar.
You can also search at Alexa's homepage to find the traffic rank for any site. As
I write this, GoingWare's is about 90,000, up from 180,000 a few months ago when I
kicked off my marketing campaign.

A word of warning: all the PageRank plugins as well as
the Alexa Toolbar work by reporting the URL of each page you visit to a server.
If you're concerned about your web surfing privacy, don't install them.

Alexa calculates its Traffic Rank from the tallying up the number of times
its its toolbar reports visiting a site. If a lot of people who have the toolbar
visit your site, your traffic rank will increase.
Not many people use the toolbar yet, so the statistics reported for low traffic
sites are not reliable: the traffic rank for a low traffic
site can be greatly affected by just a few people. If you install the Alexa Toolbar or
SearchStatus then spend a few days working vigorously on your own website, your traffic
rank will skyrocket because of your own activity.

I have a popular
PageRank 5 page that's in the top ten for a couple keywords
where all the other top ten pages are PageRank 6 and above. For one keyword
it's number eight where number nine is PageRank 6 and number ten is PageRank 7.

The reason I think is that people like to click it in the search
result listings, and (I'm pretty sure) they don't then just hit their Back
button. I'm pretty sure all the search engines notice whether someone
stays at a page they've clicked on rather than hitting Back and doing
more searches.

How? The URL in the search results is often a redirect rather than the
actual destination link. For some engines, this is the case for every
search result, Google just does this for a small sample. Hover your
mouse over a search result that turns up one of your pages - does your
page's URL show up at the bottom of the browser window, or some
funny-looking long URL at the search engine's domain?

This won't happen often at Google, but if you check each time you search
you'll see it eventually.

What happens when you click the link is that it takes you to a server
that records which search result you clicked on, and then sends you on
to the destination page. If the search engine notices that a particular
page gets clicked on more than others above it, the search engine ranks
that page a little higher. Similarly, pages that don't get clicked on move lower.

This yields my first advice for White Hat SEOs:

Use <title> tag text
that will make searchers want to visit your page.

Look over the most important pages on your site. The text in the title
tag appears at the top of the browser window, not in its content area.
Don't confuse it with the <h1>
tag, which appears in the content area of a
window. Your page's link text in a search result listing will be whatever is in your
<title> tag.

There is more that you can do though. Try searching for one of your
page's exact URL in Google. You'll see a page showing the title, a
snippet of sample text, and some links where you can find the number of
backlinks and so on. My question for you is, is that sample text what
you want people to see?

When I realized that this sample text for many of my pages was simply
"Copyright 2005 Michael D. Crawford. All Rights Reserved." I got a clue
and did the following:

Carefully compose one or two compelling sentences describing the page
and place it in the very first<p>
element after the <h1> element. Place the exact
same text as the content attribute of a
<meta name="description"> tag in your page's
<head> element.

When no other sample text seems more suitable, most search engines will
use either the meta description or the first paragraph after the
<h1>. Here is this page's meta
description:

Try searching for the URLs of a few of your own pages. Are the
results what you want people to see? Are they the best they could be?

Some people don't like to use <h1> tags.
These people don't like to get
search engine referrals. Instead, the prefer to use
<p> text that has
been made larger either with a <font>
tag or CSS. You see, they prefer
controlling the appearance of their pages to making money from their
website. This leads me to:

Use an <h1> tag on every page on your site.
Usually you want its text to be the same as your <title>
tag's. If you don't like how <h1> text looks, use
Cascading Stylesheets to control its appearance.

Your sample text can be as long as about 160 characters, with the exact amount
being different depending on the search engine. You don't have to use
all the available space, but use most of it. If it's really short,
the search engines will fill in the available space with some randomly selected text
from your page, which might not be appealing. Make your case as close to the beginning
as possible, as sometimes the end is cut off and replaced with other text from
your page.

Copy the following ruler into your page below your meta description tag
to measure the length of its content element. It is an HTML comment so it won't
show up in your page:

If you're not able to compose appealing page intros, consider hiring a
professional ad copywriter. Certainly post improved drafts as you think
them up.

Once I realized the importance of all these, I pulled a couple all
nighters implementing them all over my site. As a result, my traffic
skyrocketed, with traffic to some of my pages multiplying five to ten
times, with many of my pages moving up significantly in the search
engine results.

My next two pieces of advice are much harder to achieve, but are the most
important:

Have content that's worth your visitors' while to visit. Otherwise they
will just hit their back button.

Now, I'm sure if we had the answer to that question, we could all
retire. Yet it still seems to be news to many folks. It's worth your
while to labor over your content over a long period of time, posting
improved versions as you're able to revise them.

A corrollary to this is:

Post each page's most compelling content Above The Fold.

That is, capture your visitors' attention and convince them to stay
without them having to scroll their browser windows. If you use a hi-resolution
monitor, either shrink your browser windows or test with a lower resolution to
judge the effect of your pages' "above the fold" content.

GoingWare's site was originally meant to advertise
my
software consulting business.
Someone gave me this advice years ago and it made a huge difference to me. One
result of it was that I was able to work in
my profession throughout the
economic downturn, while many of my colleagues either had to sell or
even lost their homes and had to move back in with their parents.

I have continuously applied this advice with each redesign of all my pages.

And this leads to, finally:

Do not use "Intro" pages. Give your visitor what they want to see when
they first visit your homepage.
Never, ever, make them sit
through a Flash intro.

One sure sign of a website designer who doesn't understand how to make
your website earn money for you is that they recommend a Flash intro for
your site. My problem is not with Flash, however: some sites have static
graphic homepages, perhaps with a
great big logo that you click to
enter. You will find many of the visitors you might have otherwise
retained will instead hit their Back buttons.

The search engines will see that they returned, when the visitor you just
lost either performs a new search or clicks a different link in the
original search result page. Your position in the
search results will then get a little worse.

Good advice for designing your homepage is
Jakob
Nielsen's
book Homepage Usability. In it, he has screenshots of a hundred
homepages from prominent websites, with meticulous critiques.

Now, if you try to apply my advice to every page on your site
all at once, you will quickly go mad. Instead, work on your most
important pages first: your homepage, then a few of your most
popular content pages. Wait to apply it to your less significant pages
until you get free time. After a little while it will become very easy for you
to apply these techniques to your new pages.

Try Googling the URLs of your ten most significant pages. Does Google
show what you want people to see? Now look at each of the pages. Will they
capture your visitors even if they don't scroll their windows?

Once you can say "yes" for your top ten pages, you will fare much better
in the search engine results the next time they update their indices and your
website revenue will increase significantly.

I know this very well.
I Bet the Farm
back in May of 2005 by taking a lot of unpaid time off my consulting work to
work on my site, and as a result of the above and a lot of other things
I did, in July I quit my job for AdSense.