Leo Reading

Software Engineer and Problem Solver From the USA

SEO Displaying posts tagged with SEO

A robots.txt file is important because it tells search engine crawlers what they should and should not crawl. Unfortunately, they don't always listen, but we should take advantage of the ones that do! In today's world, websites aren't simply a collection of static pages: There is ever-changing and new content being posted, and what's there one day may well be gone the next.
Before I begin, I should mention that a lot of this was inspired by Ben Foster's Fabrik.Common library, especially the SiteMap generator. I took a lot of the same principles and applied them to a slightly different use case, and also abstracted most of it to a class library to avoid copying/pasting every time I need to publish a website.
Robots Txt Generator
The robots txt generator was made into a class library so it can be reused across many applications. It has ...