Sometimes webmasters want to take their site offline for a day or so, perhaps for server maintenance or as political protest. We’re currently seeing some recommendations being made about how to do this that have a high chance of hurting how Google sees these websites and so we wanted to give you a quick how-to guide based on our current recommendations.

The most common scenario we’re seeing webmasters talk about implementing is to replace the contents on all or some of their pages with an error message (“site offline”) or a protest message. The following applies to this scenario (replacing the contents of your pages) and so please ask (details below) if you’re thinking of doing something else.

1. The most important point: Webmasters should return a 503 HTTP header for all the URLs participating in the blackout (parts of a site or the whole site). This helps in two ways:

a. It tells us it's not the "real" content on the site and won't be indexed.

b. Because of (a), even if we see the same content (e.g. the “site offline” message) on all the URLs, it won't cause duplicate content issues.

2. Googlebot's crawling rate will drop when it sees a spike in 503 headers. This is unavoidable but as long as the blackout is only a transient event, it shouldn't cause any long-term problems and the crawl rate will recover fairly quickly to the pre-blackout rate. How fast depends on the site and it should be on the order of a few days.

3. Two important notes about robots.txt:

a. As Googlebot is currently configured, it will halt all crawling of the site if the site’s robots.txt file returns a 503 status code for robots.txt. This crawling block will continue until Googlebot sees an acceptable status code for robots.txt fetches (currently 200 or 404). This is a built-in safety mechanism so that Googlebot doesn't end up crawling content it's usually blocked from reaching. So if you're blacking out only a portion of the site, be sure the robots.txt file's status code is not changed to a 503.

b. Some webmasters may be tempted to change the robots.txt file to have a “Disallow: /” in an attempt to block crawling during the blackout. Don’t block Googlebot’s crawling like this as this has a high chance of causing crawling issues for much longer than the few days expected for the crawl rate recovery.

4. Webmasters will see these errors in Webmaster Tools: it will report that we saw the blackout. Be sure to monitor the Crawl Errors section particularly closely for a couple of weeks after the blackout to ensure there aren't any unexpected lingering issues.

5. General advice: Keep it simple and don't change too many things, especially changes that take different times to take effect. Don't change the DNS settings. As mentioned above, don't change the robots.txt file contents. Also, don't alter the crawl rate setting in WMT. Keeping as many settings constant as possible before, during, and after the blackout will minimize the chances of something odd happening.

The term "piracy" implies that the wide availability of unauthorizedcopies of copyrighted content is the result of bad actors preying onthe legitimate market. But history teaches us that it is primarily aresult of market failure, the unwillingness or inability of existingcompanies to provide their product at a price or in a manner thatpotential customers want. In the 19th century, British authors likeCharles Dickens and Anthony Trollope railed against piracy by Americanpublishers, who republished their works by re-typesetting "earlysheets" obtained by whatever method possible. Sometimes these workswere authorized, sometimes not. In an 1862 letter to the Athenaeum,Fletcher Harper, co-founder of American publisher Harper Brothers,writing in reply to Anthony Trollope's complaint that his company hadpublished an unauthorized edition of Trollope's novel Orley Farm,noted: "In the absence of an international copyright, a system hasgrown up in this country which though it may not be perfect stillsecures to authors more money than any other system that can bedevised in the present state of the law.... We cannot consent to itsoverthrow till some better plan shall have been devised."

America went on to become the largest market in the world forcopyrighted content.

That is exactly the situation today. At O'Reilly, we have publishedebooks DRM-free for the better part of two decades. We've watched thegrowth of this market from its halting early stages to its robustgrowth today. More than half of our ebook sales now come fromoverseas, in markets we were completely unable to serve in print.While our books appear widely on unauthorized download sites, ourlegitimate sales are exploding. The greatest force in reportingunauthorized copies to us is our customers, who value what we do andwant us to succeed. Yes, there is piracy, but our embrace of theinternet's unparalleled ability to reach new customers "though it maynot be perfect still secures to authors more money than any othersystem that can be devised."

The solution to piracy must be a market solution, not a governmentintervention, especially not one as ill-targeted as SOPA and PIPA. Wealready have laws that prohibit unauthorized resale of copyrightedmaterial, and forward-looking content providers are developingproducts, business models, pricing, and channels that can and willeventually drive pirates out of business by making content readilyavailable at a price consumers want to pay, and that ends up growingthe market.

Policies designed to protect industry players who are unwilling orunable to address unmet market needs are always bad policies. Theyretard the growth of new business models, and prop up inefficientcompanies. But in the end, they don't even help the companies they try to protect. Because those companies are trying to preserve old business models and pricing power rather than trying to reach new customers, they ultimately cede the market not to pirates but tolegitimate players who have more fully embraced the newopportunity. We've already seen this story play out in the success ofApple and Amazon. While the existing music companies were focused onfighting file sharing, Apple went on to provide a compelling new wayto buy and enjoy music, and became the largest music retailer in theworld. While book publishers have been fighting the imagined threatof piracy, Amazon, not pirates, has become the biggest threat to theirbusiness by offering authors an alternative way to reach the marketwithout recourse to their former gatekeepers.

Hollywood too, has a history of fighting technologies, such as theVCR, which developed into a larger market than the one the industrywas originally trying to protect.

In short, SOPA and PIPA not only harm the internet, they supportexisting content companies in their attempt to hold back innovativebusiness models that will actually grow the market and deliver newvalue to consumers.﻿

For instance, an Internet service provider could block DNS requests for a website offering online video that competed with its cable television offerings, based upon “credible evidence” that the site ...

You are an agent of change. Has anyone ever told you that? Well, I just did, and I meant it. Normally we stay away from from politics here at the official WordPress project -- having users from all ov...