Cache Warmer / Crawler

Overview

If you have our Magento Page Cache (or a 3rd party solution), then you need the Magento Cache Warmer / Crawler. The Magento Cache Warmer crawls your Web site to ensure that pages are always at their fastest. It crawls your Web site with concurrent requests for the fastest crawls possible. Your old cached pages will automatically refreshed by the crawler to ensure visitors see the most recent version of your site as possible. Crawls use special techniques to reduce bandwidth and reduce CPU usage. If you want to ensure as much as your site is cached as possible, then this is the extension for you.

What type of stores are benefited from the Magento cache warmer?
Any store that wants to improve its cache hit rate. Cache hit rate is the number of requests to your site that are uncached versus cached. The higher the hit rate the faster your site. You may have the fastest cache solution in the world, but if your cache hit rate is low, then your store will still be slow. Stores with a high number of unique pages compared to their traffic will tend to have a low hit rate and as a result caching solutions are not effective. The Magento Cache Crawler solves this by crawling and pre-caching the pages on your site, which will increase the cache hit rate. The end result is an overall faster site.

Features / Benefits (depends on checkout options)

Efficiently crawl your Magento store to warm your caches

The crawler will only access each page one time during a crawl session to ensure
there is no duplicate effort. In addition, pages your users cannot access (disabled products, children of configurable products)
will not be crawled.

Crawl all your important pages

Crawl CMS pages, catalog pages, and product pages. When used with the Full Page Cache,
all your layered navigation filter options, deep catalog pages, searches, and 3rd party extensions
will be crawled without any further configuration.

Ensure that server load on the server does not become high

A crawler increases server load in exchange for faster page speed. If there is lots of things happening on the server, then it can create high server load. The crawler can detect this and automatically pause itself and automatically restart when the server load reduces. This ensures that the crawler does not compete with user traffic for server resources!

Crawl as quickly as possible using multiple crawler threads

Multiple threads will be used which can reduce the total crawl time by 10 times or more.

Crawl as logged as logged in users (customer group)

Do you show different content to logged in users? If so, it might make sense to crawl as a logged in user so their cache can be warmed. You can now easily select which
customer group you want to crawl.

Crawl out-of-schedule jobs by manually adding it

Did you flush the cache and need to run a job real quick even though it is not scheduled yet? No problem, you can easily add a manual job that will run immediately.

Don't crawl disabled / hidden / out-of-stock products

Ensure you don't waste resources crawling products that would be infrequently viewed or not viewable at all!

Filter out generated urls by url type

Easily filter out certain pages such as category pages or cms pages or product pages from being added to the queue. This is useful
if you only want to crawl category pages or certain page types.

Always view the status of crawls and manage crawls using an easy interface

Easily view past crawls, currently running crawls, and queued crawls. Pause or disable crawls. Change
the number of threads used. View the number of crawled URLs, the source of the URLs, and view performance
metrics such as the number of crawled URLs per minute.

Filter out any URL by regular expression

Easily ensure that urls that match a regular expression will not be crawled. This is useful if you do not want certain pages to be crawled.

Fully support crawling of HTTPS secure urls

Is your site running in secure mode on the frontend? The crawler will crawl your secure pages too.

Reduces bandwidth and CPU usage during crawls

When combined with our Full Page Cache there will be reduced bandwidth
and processing done when loading uncached pages loaded by the crawler process.

Flush caches prior to crawling

Flushing Magento caches prior to crawling is important to ensure that the crawler crawled
pages are the most up-to-date.

About the Developer

Very good extension
This extension is fifth one i bought from Extendware, it absolutely me me comfort.
This extension should be must have for magento.
February 22, 2015

Website is flying!
I purchased this along with Extendwares full page cache and now my website is crazy fast. The support was great and I learnt a little something about Ajax at the same time. Many thanks.
February 5, 2015

Great Product and Outstanding Support
I have had a number of extensions from Extendware and have they have bent over backwards to help us.
Any Questions or issues that we had were answered or fixed within a few hours.
Anyone looking for a full page caching extension should definitely look at their full page cache extension, it has speeded up our site so much.
June 30, 2014

Get the most out of the Full Page Cache
I’m using the Full Page Cache extension from Extendware. This extension is loading pages much quicker (when pages are cached). The Webshop is on shared magento hosting and benefits a lot from the cached pages. There are around 3000 products in the catalog, but not all of these are viewed often. Therefore it’s important to have these pages cached to keep user experience OK also for these pages.
Installation of the crawler went fine. The installation guide indicates how the cron job for the scheduled crawling needs to be setup. Not difficult, but I made a small mistake. After sending a support ticket, john guided me quickly into the right direction. It’s working fine, pages are crawled now two times per day, as configured.
The full page cache crawler is good additional extension to benefit most from the Full Page Cache !
June 25, 2014

My 3rd extension...
This is my 3rd extension I have purchase from Extendware and they extensions quality is problem free.
With Extendware you can be confident with your investment.
Now I'm one loyal customer.
June 1, 2014

A good solution
This is a good solution if your having issues with the Magento native cache invalidating with custom updates. Extendware did a great job installing the extension to meet our needs. John and his team were responsive to on-boarding issues and questions after the install was done.
October 31, 2013

Awesome - Continual speedy cached pages....
Really essential if your using the full page cache -
We have a number of stores on the same install - so we do have edits going up and sometimes have to refresh our cache - The crawler helps us to know that our pages will soon be crawled/cached, so that our sites continue to be super speedy for the end-user ( extendware full page cache is great...)
We also use the tagged pages addon - which has proved to be really nice - adding layered nav urls etc. to the crawling schedule, that have been viewed more than three times... pretty great functionality, to decide what pages are popular and add them to the crawling cue...
Support is best we have had.
Thanks!
August 21, 2013

I just started using this extension with the full page cache extension. The two go hand in hand. The crawler and the page cache extensions are awesome. Highly recommend you get it if you want your magento store to fly. You have no idea how much faster my store is now. You get way you pay for, remember that. It will be money very wisely spent.
November 15, 2012

I have been using this crawler that is very good complement to their Full page cache extension. Simply great extension, with the new improvements the server resources have almost minimal impact. Easy to install, easy to manage in any case this is an excellent tool as the visitors always visit the cached pages (fast!) i set it up so it crawls on off peek hours and next day clients can access the cached pages.
Great support they always answered my questions.
Louis
October 17, 2012

I have been using this crawler that is very good complement to their Full page cache extension. Simply great extension, with the new improvements the server resources have almost minimal impact. Easy to install, easy to manage in any case this is an excellent tool as the visitors always visit the cached pages (fast!) i set it up so it crawls on off peek hours and next day clients can access the cached pages.
Great support they always answered my questions.
Louis
October 17, 2012

Was this helpful?Yes0No0http://answers.magentocommerce.com/answers/4643-en_us/product/12634/question/148289/undohelpfulness.djs?format=embeddedhtml&sessionparams=__BVSESSIONPARAMS__&return=__RETURN__&innerreturn=http%3A%2F%2Fanswers.magentocommerce.com%2Fanswers%2F4643-en_us%2Fproduct%2F12634%2Fquestions.djs%3Fformat%3Dembeddedhtml%26scrolltotop%3Dtrue%26expandquestion%3D148289&user=__USERID__&userdisplayname=__USERDISPLAYNAME__&authsourcetype=__AUTHTYPE__&submissionparams=__BVSUBMISSIONPARAMETERS__&submissionurl=__BVSUBMISSIONURL__

Hi. We are using a hardware solution for caching (varnish). This is really fast (<100 ms) as long as the caches are warm. The thing is, that we deliver slightly modified output depending on the USER AGENT. So if a Google Bot is coming around we ha a different output of our product pages with special meta information. Our Cache is configured to save a separate cache for each USER AGENT. The question now is, if your crawler can spoof the user USER AGENT to warm each cache?

Was this helpful?Yes0No0http://answers.magentocommerce.com/answers/4643-en_us/product/12634/question/393980/undohelpfulness.djs?format=embeddedhtml&sessionparams=__BVSESSIONPARAMS__&return=__RETURN__&innerreturn=http%3A%2F%2Fanswers.magentocommerce.com%2Fanswers%2F4643-en_us%2Fproduct%2F12634%2Fquestions.djs%3Fformat%3Dembeddedhtml%26scrolltotop%3Dtrue%26expandquestion%3D393980&user=__USERID__&userdisplayname=__USERDISPLAYNAME__&authsourcetype=__AUTHTYPE__&submissionparams=__BVSUBMISSIONPARAMETERS__&submissionurl=__BVSUBMISSIONURL__

Login

You are using an outdated browser

We built Magento Connect using the latest techniques and technologies.
This makes Magento Connect faster and easier to use.
Unfortunately, your browser doesn't support those technologies.
Use the links below to download a new browser or upgrade your existing browser.