What is Robots.txt?

The robots exclusion protocol, or robots.txt is a text file that you can use to instruct search engine bots on how to crawl and index pages on your website.

Where is the Robots.txt File Located on my website?

The robots.txt file is normally located at the root of your site:

http://www.yoursite.com/robots.txt/

The robots.txt file indicates to Google which parts of your site you don’t want accessed by search engine crawlers. Sometimes site owners will block ridiculous things in their robots.txt file, so it is an essential check to see what is being blocked.

Why is the Robots.txt file important for SEO?

Unfortunately it is all to common for small businesses and even larger businesses to block parts of their site unintentionally using the robots.txt file.

By doing this you are effectively saying to Googlebot: “Don’t crawl my site!”. This then means your pages will not be indexed in Google and you can’t rank for your organic keywords.

How can I stop search engine bots crawling my site?

If you see the below in a robots.txt file, this will instruct all robots to stay OUT of the website. From a search perspective, this is something that you generally don’t want. (Unless you are Louis Theroux, or you are creating a test site that you don’t want Googlebot or any other bots to see.

To stop search bots crawling your site enter the following in your robots.txt file:

User-agent: *
Disallow: /

User-agent: * ----> This command applies to all robots.
Disallow: / ----> This command instructs the robot not to visit any pages on your website

How can I monitor search engine bots crawling my site?

By downloading the Log Files from your web server you can analyse how search engine bots (typically Google for SEO purposes are interacting with your site). Using a program such as Screaming Frog’s Log File Analyser you can also visualise quickly which bots are visiting your website and understand whether you need to block any of them.

Navigating Responsibly is a beautiful Long Form content piece that is pretty awesome in my opinion. As you scroll down the page, the different sections of the site burst into different layers of animation, shareable quotes and high definition images. The different quotes on the page are clear and are easily shareable with animated Twitter, Linkedin and Facebook sharing buttons. It really shows that a lot of thought has gone into creating this thoughtful content piece.

Overall, quite simply, this is a beautifully executed piece of content.

For me, when you are creating Long-form content, it should achieve three things:

1.Functional

It should be functional in terms of its sharing capacity: Make no mistake: You are creating this content to get social shares, juicy links and substantial brand coverage. If your content isn’t functional in terms of its shareability, then there will be little point in getting thousands of visitors to the content if you have little else to show for it afterwards.

2. Convey a simple, clear message to the visitor. What else?

Communication is key for this type of content. What is the end message, what is the story behind the creation and why are you taking the time to make this content in the first place? These are all questions that any visitor to the content would want to know the answers to. The piece must convey a key message to the end user.

3. Visually Impressive and above and beyond that of ‘normal’ or shorter form content

This message can often be amplified greatly by creating a user experience that goes above and beyond what many people expect to see within that industry or niche. This can be achieved by fancy-pants visualisations, high definition photos, interactive bits, videos and so on. This is the icing on the cake if you like, but to get that initial “Wow this is so fricking awesome” feeling from your visitors, you need to work on this aspect the most. (Unless you have a damn good idea).

Common Stumbling Blocks

This is all well and good, but sometimes if you go too far on the visual side of stuff then that can affect the core functionality of the content. The key to creating this type of content is to get the balance just right.

I think this content piece does a really good job of balancing these two functions of this type of content. Generally it is not easy to get the balance right, and depending on your team structure, the content can become either too functional with no styling, or highly visually impressive with little to no core functionality. Balance is key.

What is the Donald Trump “Don’t” Billboard?

A few days ago, people started sharing a piece of content described as a prize winning piece of work and credited it to The Economist. Everyone had a good laugh because it was another piece of content that had a poke at one of the most controversial presidential candidates ever. We have already seen other pieces of hilarious content this year that have been specifically targeted to jump off the Trump-mania bandwagon. The thing that made this one good is it was so simple in concept, but carried a powerful message. The viral content featured a red background on billboard and used a font similar to the Economist.

The Epic Twist to the story: It was produced by a London Agency

Like all good stories, there is a twist: The Economist didn’t actually produce this billboard. It was made by an advertising agency based in London. It was the work of @typechap who works at AML Group. BOOM! Well done guys.

Warning: If you think insects are gross then this post is not for you!

In my opinion, this is certainly one of the most unique pieces of interactive content to come out this year. This content can be accessed here on the Microsculpture.net website. This piece is essentially macro photography taken to the next level.

Expert photographer Levon Biss worked with Oxford University and Entomologists to create what is one of the most fascinating visual insights into the insect world that is available on the internet. When you access the content you can zoom into micro details on each of the insects. The video below explains all:

The beauty of this idea is that it is very simple, but one that has been taken to the absolute extreme and has produced something artistic and creative out of something where many people wouldn’t have thought there was any artistic value – let alone a full blown interactive content piece.

This type of content works because it has the power to shift people’s expectations: Initially when I first saw this piece, I thought it was a bit gross; lets be honest, insects aren’t everyone’s cup of tea. But when you zoom in so close, you start to see a lot more detail in the insects that you have never seen before, and it twists your perception of what you were expecting to see. Far from being boring, the insects are colourful and have levels of details that you just don’t expect to see. Pretty cool stuff!

Brighton SEO – The Landmark moment

It must have been an exciting moment for Kelvin Newman, organiser of #BrightonSEO, when he realised that the event he started in a room above a pub a few years ago has now outgrown the Brighton Dome. This event is his baby, and it has now graduated from the Brighton Dome due to its intense popularity. This is a huge moment for SEO and for the event itself. The question that now has to be asked is how long will Brighton SEO stay in Brighton? Will Brighton SEO grow so big that it has to be held outside of Brighton? That would indeed be interesting:

Part of the secret sauce that makes Brighton SEO so much fun is the very fact it isn’t in London. For London Agency SEOs it is a chance to escape the big smoke and eat Fish and Chips on the Brighton Pier.

Yes, Brighton SEO is a big deal

Brighton SEO is starting to get some mainstream acknowledgement from London City type papers:

Having been to Brighton SEO a few years in a row now, I have been going long enough to remember when it was just a few people walking around in the lobby hall at break time with the Screaming Frog guys playing Table Football in the corner. This year however, it was more like getting into Canary Wharf tube station at 5.30pm in the break. It was THAT busy. Brighton SEO has, it seems, now increased its Citation Flow (Trust Flow was always high) and has evolved into a big, bustling event. The event was even streamed on the Internet this year, which I managed to accidentally Tweet Bomb.

Goodbye Brighton Dome, thanks for all the good times

The Brighton Dome has served us well, the last few years the venue has hosted some of the best talks on SEO from some of the best SEO speakers in the industry. I hope that this will continue and the spirit of Brighton SEO continues. (With Kelvin in charge, I am sure it will).

Matthew Barby, Kirsty Hulse, Jon Earnshaw are just a handful of the Superstar Speakers (here we go, Philip) that have over the years consistently shared with us some of their top tips for getting to the top of our SEO game. And indeed, we did get to the top of our game. Seeing younger speakers this year showed that the next SEO generation is starting to break through and have a big impact in the SEO world.

Meanwhile, Google is still constantly raising the bar

Penguin and Panda are now part of daily-accepted hurdles of the digital landscape that SEOs are tasked with dealing with. Far from putting SEO agencies out of business, by introducing Penguin and Panda updates, Google has actually helped SEO agencies bring in more business and create more awareness of the website issues that Google doesn’t like and will penalise. The SEO industry has, as a result, turned from being something that sneakily tried to beat Google at its own game, to something more akin to medical professionals; SEOs now diagnose the ‘health’ issues with your website, and suggest a remedy in the form of a long term strategy.

SEO Agencies and Freelancers have been forced to become slicker, more analytical, and increasingly focused on returning ROI for the client. SEO is by its nature very hard to put specific numbers to, but Forecasting for SEO is now no longer something that agencies shy away from, it is something that they provide in the pitch stage to illustrate the confidence of their strategy.

The importance of user intent has also been highlighted in Google’s updates this year, and has again caused SEOs to up their game certainly in regard to creating content and understanding the importance of user experience. If you are churning out lots of 500 word articles for your client every month, then quite simply, you are missing the point. The game has changed significantly this year, and Google Bot is becoming increasingly more sophisticated when it comes to recognising the hallmarks of what it perceives as so-called “Great content”. Yes, you can still trick Google into thinking something is high quality content, when it isn’t, but seriously what is the actual F***ing point, when it would just be quicker to create something decent in the first place!

The people who have been affected the most are the people who tried to ‘Game the SERPs’. Unfortunately clients who were small businesses that hired people in the past who were unscrupulous SEOs may have been hit the hardest. Yes, this may be considered by some to be fun, but ultimately none of us want to live in a world which is spam filled. Spam is annoying.

Why as SEOs, we need to thank Brighton SEO

Brighton SEO has done a lot to improve the image of SEO in general. The consistency of the event has allowed SEO to shift its darker image, and people have come to understand SEO as one of the most forward thinking marketing disciplines in the industry.

When you go to £400+ per ticket marketing events, you get fancy food, you get great speakers, you get a nice posh venue, but there is always the feeling in the back of your head that something is missing. That is because the community spirit that exists at Brighton SEO is not present at these other events.

The lessons from Brighton SEO for other conferences

The camaraderie of Brighton SEO is what makes it special, meeting up with old friends and colleagues is always fun, and there is no excuse not to attend seeing as it is free – which is just one of the reasons we keep going back. Again, and again, and again.

In a world where marketing conferences and even award ceremonies costs hundreds of pounds to get tickets for, it would be very easy for Kelvin to monetise Brighton SEO and charge £400 quid per ticket, but he hasn’t done that, and 1500 of the tickets are still free (if you can get one in time!). This means that often whole teams end up going to the event and it becomes a team outing.

SEO is no longer a ‘dark art’

SEO was a mysterious beast a few years ago. Ask any marketing salesmen who was involved in digital sales for over 10 years and they will inevitably smile at you when mentioning SEO, and they will say “Ah yes, the dark art!” Then the inevitable hat colour discussion will ensue. Don’t get me wrong, SEO is still not exactly the most straightforward discipline to understand, but it is one that with help from events such as Brighton SEO has been able to open its doors and show that the techniques that are used in SEO are thoughtful, intelligent and arguably more ROI focused than many other marketing channels.

SEO nowadays has cleaned up its image. It is now more about ideas, creativity, productivity, process and improving your technical ability. Yes the forums, ‘churn and burn’ sites, high PR links, get rich quick schemes, ‘guaranteed page 1 results’ are all still there, but are being increasingly scorned upon as representing the lower quality aspect of SEO. Events such as Brighton SEO have done a lot to showcase to the world the best side of SEO.

If you asked someone in the street about SEO, 9 times out of 10 they would probably give you a blank expression, yet it is one of the most relevant and important things that you can learn to help promote your website or business.

So why has Brighton SEO and SEO in general become SO popular, particularly in the last year?

1. SEO has grown up – It is taken very seriously by all types of businesses now, and they can’t invest in SEO quick enough. 2. SEO is becoming more creative than ever before. Agencies are redefining what is possible in terms of content and getting better results than they ever have done previously. 3. SEO has evolved into a multi-skilled discipline. You can’t learn SEO without learning a little about some of the other channels interact and work. 4. Professionals in SEO are at the top of their game. – SEOs are by their nature fond of making complicated tasks as simple as possible. SEOs are generally among the first people in the industry to talk about new tools that can help improve a process. 5. Companies like to have a presence at Brighton SEO. It is an event that companies will gladly send their employees to because it is free and it is well known for having great speakers.

Final Word

Thank you Kelvin for another fantastic Brighton SEO. I think every SEO is looking forward to seeing what is in store for the next event. Sure it will be amazing!

It is not often you get to see a Gold Ferrari in London. That is exactly what I managed to spot on the weekend though. Customised Ferraris are pretty commonplace in London, especially in highly affluent areas such as Knightsbridge in West London.

Recently the trend with the super rich has become to being as absurd as possible with your colour choice. No longer is it just good enough to have a Ferrari. A statement needs to be made to separate your super car from the next super car parked down the street. What better statement than painting your car gold! (Before you ask, unfortunately as far as I can work out, this isn’t Richard Baxter’s new car).

After a little detective work it turns out this car belongs to Iraqi Kickboxing champion Riyadh Al-Azzawi.

What is a Log File?

A Log File contains the records of hits from different user agents that the server receives. The data in the log file contains data such as the time the hit or request to the server was made, the IP address, the URL requested and the user-agent used.

Why is Log file Analysis useful?

Log File analysis is one of the best ways to understand exactly how different user agents are crawling your site. Log File analysis is particularly useful when it comes to understanding how much Crawl Budget is being wasted, and crucially, what URLs this is on. Any accessibility errors will also be illuminated during log file analysis as well as any other crawl deficiencies. For example, if you suspect Googlebot is ignoring some pages where thin content is potentially an issue, but you need to prove this to your client, this is where Log File analysis can be used to categorically prove this through our old friend: Data.

For the purposes of SEO, we will mainly be filtering the Googlebot User-Agent. (Other User-Agents are available). To perform a decent analysis you are going to need around 60-120,000 rows of Excel data.

The Anatomy of a Log File

Now you know a little about the basics of log file analysis, it is now time to show you the different sections of a Log File. Log Files are consistent in the fact that they will almost always include the following:

Server IP

Date and Time

Method (GET / POST)

Request URI

HTTP status Code

The User Agent

Log Files may contain other information, such as the host name, client IP address or the bytes downloaded.

Crawl Budget: What is it exactly?

Crawl budget is the allocation of pages given to your website by a search engine (such as Google) each time it visits your site.

How is Crawl Budget determined?

Crawl budget and allocation is based on the authority of your site. In the olden days this was determined in part by PageRank. Essentially, the more authority your site has, the more URLs will potentially be crawled.

Even if your site has a large Crawl Budget, Google may choose to ignore certain sections of your website if it sees that you are producing content that is thin, or low quality on a large scale.

Log File Analysis Tools: Screaming Frog Log File Analyser & Splunk

The Screaming Frog Log File Analyser is one of the most user friendly ways to analyse your Log Files. As this product is made specifically for SEOs it is the one I recommend above all others, although if you are looking for other alternatives there is always Splunk, which is also reasonably user friendly. (Although the option to simply drag and drop a server file from any server such as Apache, ISS or NGINX on to the actual tool for fast analysis is not an option).