Crawler Face-off: Xenu vs. Screaming&nbspFrog

Over the past couple of years, we've seen a number of new web-based SEO crawlers (including our own Pro App here on SEOmoz). They're great tools and can provide a lot of insight, but sometimes you've just got to get down and dirty into a big site, and you need a desktop crawler application. For the past few years, that's pretty much meant Xenu's Link Sleuth. Recently, a new entrant has thrown its hat into the ring – Screaming Frog SEO Spider.

I was amazed, on social media, how many people still hadn't heard of Screaming Frog. Actually, a few people still don't know about Xenu. So, I thought it would be a good time to put both programs through their paces.

Xenu's Link Sleuth

In this corner (that doesn't really work when you can't see me pointing, does it?), we have Xenu, a PC-based desktop app that bills itself as a tool for checking broken links. Over the years, enterprising SEOs have found it's capable of a lot more. For such a popular tool, it's funny how little we know about the creator, a German [engineer?] named Tilman Hausherr. All I can tell you is that he wrote Xenu because he was bored, he doesn't want your money, and he really dislikes Scientology.

Pro: Xenu is Free

Let's start with a few pros and cons. You really can't beat that Xenu is completely free. We all owe Tilman a hearty "thank you" for putting it out there over a decade ago.

Pro: Xenu is Fast

While a large-scale site can take a while to fully crawl, Xenu is a machine, and it's an impressively multi-threaded app, considering when it was built. It can power through 1000s of pages pretty quickly.

Con: It's Not Intuitive

Xenu really takes some getting used to, and it's clearly built by an old-school software designer. The basics are easy enough, but getting into the real SEO applications of Xenu takes a bit of a learning curve. To be fair, it wasn't built for what most of us try to do with it.

Con: No Mac Version

This is the biggie for many people – Xenu only runs on Windows PCs. There's no Mac or Linux version, and the author doesn't plan to release one.

What Can Xenu Do?

At its core, Xenu really is just a link checker, churning through your site to test internal and (optionally) external links:

Since Xenu operates like a crawler, though, it can really help test crawl paths and find holes in your internal linking. Xenu also reports useful stats, like the crawl level, outbound links from a page, and inbound links to a page. It also returns TITLEs and META descriptions, which can be handy at-a-glance.

One really useful feature in Xenu is that, once you find a broken link (page or image), you can just right-click on it, select "URL Properties", and you'll get a pop-up like the one below showing you all the pages linking to that broken file:

Xenu is a work-horse, and it's still a great tool for churning through links fast. It's only real fault is that we've tried to push it beyond what it was designed for, and it wasn't really designed for SEO.

Screaming Frog SEO Spider

That's where Screaming Frog comes in, a desktop crawler specifically designed for SEOs by a UK search marketing firm. Because it's built specifically for SEOs, it has a lot of bells and whistles that Xenu doesn't. Let's start with the quick list of pros and cons.

Pro: It's Built for SEOs

We'll dive into this one below, but Screaming frog has been built by SEOs for SEOs, and that fact is obvious as soon as you launch the application.

Pro: It's User-friendly

I found Screaming Frog very easy to dive into. The advanced uses will take some time to discover, I'm sure, but any experienced SEO should be able to get up and running on it quickly.

Pro: It Runs on Your Mac

Screaming Frog supports both PCs and Macs. If you have a Mac shop, that's a big one.

Con: It's Not Free

Ok, here's the down side. It's £99 per year (that's about $162, currently) to run the full version of Screaming Frog. The free version will only crawl 500 URIs. The free version is nearly fully functional, though, so you can get a good idea if it's worth the price for you.

What Can Screaming Frog Do?

On the surface, Screaming Frog doesn't look too much different than Xenu:

Like Xenu, Screaming Frog tracks the crawl level, internal links, outbound links, TITLE tags, and META descriptions. It also adds:

HTTP status codes

TITLE tag length

META description length

H1, H2 tags and lengths

Canonical tags

Many of these elements are also broken into their own tabs at the top, so that you can easily focus in on problem areas. You can right-click on any URL for many more options, including checking indexation in Google, Yahoo, or Bing and checking backlinks in Open Site Explorer, Yahoo Site Explorer, or Majestic SEO:

The paid version includes one incredibly useful feature – the ability to custom filter and search HTML source code across a site. Digging into all of the features is beyond the scope of this post, but Branko Rihtman has a nice Screaming Frog review over at SEO Scientist. If you want to dig deeper into Xenu, Tom Critchlow had a good Xenu recap here on SEOmoz last year.

And The Winner Is...

Sorry, I just can't do it. I've loved Xenu (the app, not the galactic overlord) for years, and even if it weren't free, I'd have gotten my value out of it many times over. I'm excited about Screaming Frog and what it can do, though, and I'm especially excited to see new products and innovation finally coming to desktop crawlers. This is a win-win for technical SEOs everywhere, and I'm looking forward to the evolution of both desktop and web-based crawlers.

it is somehow funny that you wrote this post about Xenu and Screaming Frog this week, as I rediscovered both this week due to audit jobs; maybe I was "incepted" by your question on Twitter.

I have to agree with you about the "not declared" winner. Even though Xenu Sleuth is surely a powerful tool, maybe the fact it was not really created for internal SEO analysis purposes makes of it a little bit harder to learn. I use it a lot, especially after the last year Tom's post about it and thanks to all the posts Distilled dedicates to it (last, but not least, this recent post by Mike Pantoliano about Site Structure Audit).

But if you need something that delivers you a very friendly and complete output, then Screaming Frog is the right choice. It has a cost (but it is a one time payment and not so high), but if you have to check up small sites (under 500 pages) you can use it for free.

Finally, I won't forget a third alternative: IIS SEO Toolkit. Powerful and multilingual, even though should have to update the suggestions it gives (still refererring to somehow dated factors' importance). Richard Baxter and Richard Butler have dedicated some interesting posts to it on SEOgadget (I suggest you to start from this one).

P.S.: and yes, the SEOmoz Crawler is fantastic. Shame you have to wait sooooo long before it delivers you a crawl (looking at the SEOmoz devs team)

We have used Xenu for a long time and love it still to this day. It just didn't quite do enough for us from an SEO perspective as time went by and hence we built the SEO spider.

Originally this was just built and used by us internally for our SEO projects, but we decided to release it and see that others thought as it saved us such a significant amount of time. It's fairly well known in the UK, but has taken a little longer to get picked up in the States.

Anyway, appreciate the post. If anyone has any feedback, love to hear it. We have a very long development list of features to add to the tool, so keep an eye out as there is much more to come.

I've fully moved over to the paid version of Screaming Frog, and the team have been great at adding little suggestions - so thats putting it ahead of Xenu.

Ironically our corporate firewall was what prevented me using Xenu to start with, as newer ones falsely claim its a virus! Hence my need to try out something new, and luckily found Screaming Frog - a great tool!

Thanks for chiming in on the post - love the tool so far, and look forward to where you guys are headed with it. As entrepreneurs, it seems like the best tools are always the ones we build to solve our own problems.

It is nice to see this. Using Screaming Frog felt like someone was trying to answer the limitations of Xenu, and it is nice to read that you guys were fans too. One thing that Xenu does have and I have not been able to figure out in Screaming frog is the ability to do crawl password-protected pages. I was working on a page on a dev server, and had to revert back to Xenu.

That's great, I didn't know about this toll and I'm testing it right now. Fortunately my main site didn't have any errors, but I did find that I should do about my internal linking strategy. Some pages that have hardly any value to the search engines get a lot of links.

Excellent post. I switched from Zenu to Screaming Frog shortly after a reading post that I believe Rand did covering some of the Top SEO tools. I was blown away about how much easier it was to use and that it gives amazing visibility to other key SEO factors you would want to review in an audit. Definitely improves productivity when audting sites!

I am amazed there isnt more buzz about this great tool. I highly recommend it for any inhouse SEO's - both large and small - the paid version is worth every penny. I help support several large, complex ecommerce sites on SEO and when I use this tool, leveraging filters/settings properly within the tool, its amazing how well it can handle these kind of sites.

Their support has been fantastic so far as well (just like SEOMOZ's!). I did not know what to expect but was pleasantly suprised on how professional, friendly and responsive they were. My only wish is that it could save and store the report data after it is run. I have had a couple instances where my computer has installed those darn Windows updates without my knowledge and restarted after the tool has been running for hours. Once this happens, all data is lost and you have to basically start over. Also- it would be great to have the history stored so you can see if the improvements you make from using the report data in this tool are working. That's one of the things I love about SEOMoz's crawling tool. Having the history and being able to show visual charts on increases or descreases in 404 errors, missing titles, etc has been very valuable when I have to report progress to mgmt. Sometimes it takes awhile before the impact of SEO improvements show up in traffic numbers so, being able to visually show the progress of cleanup helps mgmt understand that things are happening and to hang tight for those improvements to start reflecting in traffic and sales. Without this kind of data, their perception is that nothing is being done because traffic #'s are not increasing right away as they expect.

I am a huge fan of SEOMoz and Screaming Frog Tools and so thankful that companies like yours and SF are investing in development of SEO tools that are reliable, help improve productivitiy, and are designed with usability in mind. As a corporate SEO, it makes my life easier when training others in my organization who are "non-technical" on how to leverage these tools. Now, less reporting & monitoring falls on me which allows me to focus more time on the more complex technical SEO issues on our sites and web platforms. Both of you, please keep up the great work!

Thanks Susan, appreciate the comments. Great to hear it's making your life easier :-)

I am not sure if I mentioned this when we spoke (via our support), but we are working on a save project/crawl type feature that you can re-upload back into the tool aswell as a failsafe to avoid losing data during unexpected crashes etc.

I have used the full version for over a month now and found it briliant. Features that I would like to see added is a way to control a number of threads the Screaming Frog uses and the way to contro the number of reqests per second.

I second the request for a project save feature. I have the paid version and many of the sites I audit are enterprise level 20k+ in size. The tool is awesome, just want to make sure i can save and come back to the data as needed. Is there a timeline for when this feature will be released?

Good post Pete.. But Still my winner is Xenu.. I consider it be one the excellent Freeware helpful to all!

I consider Xenu as one of the best Freeware software and a must for a Online marketing professional. Have you tried Web Data Extractor, something similar, but not sure it is as good as Screaming Frog SEO Spider. But the issue was that too is paid one. Anyway will try out the Frog SEO Spider..

I feel something weird with the name ‘Screaming Frog SEO Spider’, but that may be the differentiating factor the company looks into for marketing it!

Thanks for this post Pete, I've been looking forward to it. After I changed to a Mac, I freaked out a bit when I realized there wasn't a Mac version of Xenu. I asked on Twitter if there was anything similar to for the Mac and no one knew of anything. I'm definitely going to be giving Screaming Frog a try. As much as I love Xenu, I really dislike having to go back to the PC just do use that app. :)

you made my day with this post. I was very excited about this post. Dan has done a good job with the Screaming Frog tool and I am excited about the improvement opportunities that will result from more vendors producing these SEO Tools.

One of my dream tool would be the merger of SEO Browser and Xenu. Screaming Frog is heading in that direction but a couple of things need to be added to make it even sweeter. But I agree, no clear winner here. Xenu in my heart, Screaming Frog in my Head.

I felt your pain at first as well. I use screaming frog on large complex websites and based on some great advice from SF- make sure you are leveraging the "Exclude" and "Spider" options in the Configuration section. By using this, you can actually set it to specific areas (smaller chunks of the site) that you can run. You can also turn off crawling of Javascript, images, etc and keep the crawler focused on pages. By leveraging both these areas, I have been able to get the tool to crawl 70k+ pages. Their FAQ section has some instructions on how to manage these configuration settings. Best of luck to you!

This is good news for me. I have several large sites 300k+ pages and while Xenu allowed me to specify a starting directory it didn't allow me the option of limiting to ONLY that directory. I could exclude urls that begin with xxxx...but that would mean manually inputing probably over 1,000 values to skip! And I end up having to run it three or four times and then having to start it over to skip certain directories that weren't necessary. So this is my question:

Can Screaming Frog allow me to specify a directory as an include only pages in this directory? If not....note to devs...I'm buying! I need to create sitemaps for several directories as many of them are larger than 50k pages. Anyone know of a better way to do this? Currently I'm working with a complete Xenu crawl of 300k urls, exporting to csv and then segmenting out the directories there and creating several url list .txt files. Then I have to use A1 Sitemap Generator which is one of the only ones I've found to import URL's from a file and output an xml sitemap file.

answered my own questions. Looks like it can't even generate an xml sitemap. Let alone images or video. Also, you can't really limit the crawler to a specific directory. It will still crawl outside that directory if it finds a link. I was hoping this would solve my sitemap creation woes...still looking for a good tool.

awesome! I'll check back in periodically to purchase once it does those things. Do you announce those kinds of things on Twitter? I'll follow you there for updates. Great to have you posting here in the blog!

I use Xenu to find check broken links and no. of pages on a website that are otherwise hidden from search engines. I like to compare no. of Xenu pages to the no. of pages indexed by Google to find any indexing issues in a website.

I don't really trust xenu when it comes to checking server response though as the no. of request time outs it gives is more than actual (may be it's coz it's running on my machine). Its reports can freak you out if you check your server performance from this tool.

Xenu generated sitemap.xml also get a little messy.

So in all I use xenu:

to check any major crawling errors

to check no. of crawlable pages in a website (to find canonical issues)

both tools are great, but tonight i have to perform a SEO audit for a client and test Screaming Frog and this tool just blow my mind : simple and beatiful interface, the tool is fast and the report is easy to read even for non expert. screaming frog team : well done :)

I am a new bee to the Link Analyzers and for a use case we have, tried running both Xenu and Screaming Frog on a public website ,both are free versions and i see a huge difference in the result set of Xenu and Screaming frog.Is it cause of licensing? Can we set how deep it can go in screaming frog?

Also I need to crawl site that needs authentication, how to achieve this with Xenu/Screaming frog?

I used screaming frog to analyze our site perunature.com. It says that there are no metatags or meta keyywords but when i check the source code I see that they are present. Any ideas on why this is happening?

All I need now is an affiliate deal with The Screaming Frog :o)That way I too can make money on that exceptional tool.I am looking into the other tools mentioned on this site, and might even make a link to their reviews when they are done.Thanks for an excellent post, as always

A long-time fan of Xenu, I must say though that it should upgrade if it wants to complete with tools like Screaming Frog. But just because it's good old Xenu, I wouldn't rule it out completely. I am sure there are people that are very used to it, who are comfortable with it and wouldn't give it up easily.On a side note, as somebody already mentioned it's not just Screaming Frog vs Xenu anymore. There's SEOmoz site crawler, the new Webmeup crawler.. It's getting really crowded in the jar :)

We've recently launched seopler.com and were selected for the Web Summit 2015 Alpga program. Our tool does everything XENU and Screaming Frog do and more. Also as our software is cloud based no local resources are consumed. Millions of pages can be crawled as an elastic server is run up for each job submitted. As one Web Summit attendee we demonstrated to quoted.

"Seopler is to SEO what Excel is to Accountants"

We offer a free account and would love feedback from the Moz community on our product.

After using Xenu for years (and loving it), I recently started using IIS Toolkit and more recently came across Screaming Frog. From a usability perspective I don't think you can go past IIS toolkit, but I must admit I'm really impressed with Screaming Frog (even the free version). Having said that, Xenu will remain on my list of tools to use, and I've even found myself running two or more of them when conducting site audits, just as a sanity check. It's interesting that from time to time I do get different results from each tool. In particular I've seen IIS Toolkit report broken links that are actually fine.

I have a problem with both when trying to crawl really big sites, 1 million+ pages, when is not possible to do it by sections or folders. I get a memory exhaustion error or similar. Does anyone know and alternative to do it?

No luck trying to find one and I'm already thinking about programming it by myself, but I'd like not to..

I saw today a 64-bit version of Xenu which should be able to (in theory) address more memory.

9. What is the maximum number of URLs that can be checked?There is no fixed number, but it seems to be above one million. The problem is that Windows XP applications have a size of 2GB max.

A 64 bit beta version is available which may or may not allow more URLs. It is based on Microsoft Visual Studio 2010. (Rename the xenu.exe that you already have installed). The 64 bit EXE file is much bigger than the 32 bit version. I suspect that I am making some config mistake with Visual C++ 2010 and that it is packing a ton of library stuff into the EXE file that isn't needed.

Are you able to use the IIS SEO Toolkit? It scales well. Totally understand your issue though.

You can increase the memory limit in the SEO spider. It comes with a standard set 512mb allocated of memory (chip rather than disk).

I haven't included it in the FAQ yet, but will explain how to do this, which will help you crawl much, much larger sites.

That said, a million URI is a lot (for a desktop program) - We are working on a few things behind the scenes to make crawling larger sites scale better and also easier to specify the crawl where the IA or structure is not particulary intuitive.

OK thanks. Would be nice to be able to do it with Screamingfrog in a more scalable way, but thinking about how, I see the difficulty on a dektop app.. if you can solve this thing would be really great.

Ahh.. Wonderful! Let's see if Screaming Frog brings a Virus Scanner Warning as well when you try to install (that's what kept me from installing it at work - admins really freak out over some misguided virus alert... *sigh*)

I didn't knew Screaming Frog neither... I find it a very good alternative for Xenu (so I don't use Xenu daily). The platform is very clearly arranged. But as you wrote, Xeno wasn't originally programmed to use it for SEO purpose...

nice post Dr Pete. Tools like this need a little more love in blogs than they get. I picked up Screaming Frog after Dave Naylor mentioned it a few times at the recent ThinkVis conf over here in the UK. Used it on every site I've touched since.

Nice post ! But in all honesty both softwares have their own quality. Like you mentioned Frog provides great insight info such as canonical tags where as Xenu doesn't

I am in love with both of these software but I do want to add a negative point of Frog which is that if you are running a big ecommerce website it tends to take longer that my grand mother's shower! It's not easy waiting in the third World country for the bathroom especially when the bloody water runs out!

Good comparison. I've been using Xenu for a few years now and I still prefer it over Screaming Frog for the sites I do work on (most sites I work on have less than 50 pages on them with exception to a few 1k+ page sites). Overall, the cost of Screaming Frog is actually one of the lower ranking reasons as to why I haven't purchased it. I find that Xenu gets me enough information and a few free web-based applications fill in the rest of the features.

It's nice to have a tool that can knock out several other tools, but sometimes having 2-3 extra bookmarks seems like the more sensible option as oppose to purchasing more software

Dr. Pete thanks for the informative post. I started using Xenu recently but will have to check out the Screaming Frog product. It's nice to see their team engaged in the discussions here. Sounds like someone needs to do a follow up Screaming Frog "Tips + Tricks" post.

Xenu is a great free Windows option but the interface is nearly unbearable. For our Macs here (The Blue Cube), we use some software called Integrity which was mentioned in another, earlier comment. Whilst it can sometimes be a little buggy, it's extremely useful and has some great options. It's at its most powerful when custom-configured to the site it's crawling.