[UPDATE 31st March 2015 - if you have re-registered according to the instructions below and are wondering why your new map on www3 has started at 0 and when it will acquire the old dots, please carefully re-read items 2 and 4 below - this is still waiting for a solution.]

If your map was stored on our “www4″ server (you can tell because you can see “www4.clustrmaps.com” or ” ’server’ : ‘4′ ” in the HTML where you embedded the ClustrMaps thumbnail map), then some major changes have taken place in the last few days (March 23-25th 2015), and we recommend that you carefully read all the items below.

Your data is safely stored*, but regrettably server www4 has now been declared ‘dead’.*(Safe storage of data is true for the overwhelming majority of users – a very small proportion of data was lost during the series of crashes that plagued us over the last few weeks, and we will know more about this over the coming weeks as we provide retrieval details – see next item).

We will provide details about how to retrieve items like your most recent visitor totals and more in a later blog posting or special page, to which we will provide a prominent link (here and in other postings) when it is ready.

We recommend that you register your URL again just like a new user would register, on our home page at http://www.clustrmaps.com or directly on http://clustrmaps.com/admin/register.php. This will mean that a fresh count will be started at 0, but once you can retrieve your earlier data (as mentioned above), then you will be able to provide a ‘footnote’ or ‘caveat’ near your new map that describes the earlier visitor data.

Paying (ClustrMaps+) accounts: If you are a paying (ClustrMaps+) user on the affected server you have two options (re-register vs refund), as follows -

You can re-register as suggested above, but it may take us a few days to get you properly upgraded to ClustrMaps+ status for the re-registered map, since this cannot be automated right now; if you are in a particular hurry, you can contact us at accounts@clustrmaps.com and be sure to mention ‘upgrade’ and tell us your registered ClustrMaps+ URL (or multiple URLs) and approximately when your subscription(s) started. Please be sure to re-read items 2 and 4 above to make sure you understand that your new maps will start from 0 and that it will take a while before you can get access to any previous data.

Refund for paid accounts: If you do not want to re-register your URL as suggested above but would instead prefer a pro rata (proportional) refund covering the outage period from March 9th 2015 to the end of your subscribed period, then please contact us at accounts@clustrmaps.com and be sure to mention ‘refund’ and tell us your registered ClustrMaps+ URL (or multiple URLs) and approximately when your subscription(s) started.

Could such a terrible crash happen again on another server? Sadly, the honest answer is “Yes”. We take great precautions, and have great backup systems, but they are not infallible, especially for a small company like ClustrMaps Ltd.

As you can appreciate, we are devastated by this unexpected loss, and the impact it has on thousands of our loyal users. We have worked very hard over the last few weeks to bring www4 back to life, but it appears that the complexity of the ‘cascading’ problem defeated our best efforts and overwhelmed the resources we could allocate. We would have notified everyone sooner but it was not possible due to a combination of (a) we were optimistic about finding a solution, as we have always done in the past and (b) there are too many users with expired email addresses, so we have to resort to this blog posting and a link to it through the thumbnail map you see at the top of this post.

We are overwhelmed with emails, so unless there is a particular dilemma you need urgently to bring to our attention, we would be grateful if you only contact us concerning refunds, as mentioned in the numbered items above.

We’d like to express our apologies once again and thank the many users who sent us their wishes and support.

[UPDATE 10, 23-March-2015 22:50 GMT: Not much progress beyond UPDATE 9 at this point (see that update below if you are a paying user) - we note that users' patience is starting to run out, so we're urgently looking into other options to try to restore the service or at least arrive at a more definitive statement of its fate. Please bear with us... thanks!]

[UPDATE 9, 21-March-2015 11:00 GMT: Refunds? It looks like this terrible interruption of service is going to be continuing for a while. We are devastated by this outage, but it has grown in complexity over the past few days, just for this specific server www4.clustrmaps.com. If you are a paying ClustrMaps+ user, and you would like a pro rata (proportional) refund, please contact us at accounts@clustrmaps.com and please tell us your registered ClustrMaps+ URL and approximately when your subscription started. Many thanks again to all.]

[UPDATE 9, 19-March-2015 00:20 GMT: Slow progress - a major data transfer and recovery operation is underway. Our data sets are massive, and require a significant transfer and recovery operation due to the multiple faults that happened on this server. This may take a few more days, I'm afraid - there were multiple underlying problems that need to be resolved, and we will provide further updates as soon as we have some more tangible progress to report.]

[UPDATE 7, 18-March-2015 00:45 GMT: Server replaced, maps running fine. In fact they have now been running for the past 8+ hours, since about 16:00 GMT 17th March, but we wanted to let it run for a while to confirm that all was OK. The map updates with the latest counts should start appearing during the next two days.]

[UPDATE 6, 17-March-2015 12:00 noon GMT: Chassis swap completed overnight... but... there's still a lot of work to do re-installing and verifying all the services... we're hoping all will be OK by tonight, but you can see from the long ongoing saga in these updates that it would be unwise to promise anything at this point...]

[UPDATE 5, 17-March-2015 00:15 GMT: Better news -- after the previous update (immediately below), things were looking like they might take a few more days, but now we're talking about hours rather than days, as a (simpler) 'chassis swap' is underway... new motherboards and controllers basically, and the shiny new disks already loaded up with all our software and data, so we're optimistic again. It's been a real roller-coaster ride, and the support of our thousands of fans and users is greatly appreciated!]

[UPDATE 4, 16-March-2015 22:45 GMT: Bad news -- well, we thought we had it solved with a double disk replacement and rebuild! Alas, there are ongoing issues with the (old) server itself, and this probably needs replacing too, which is going to mean an outage of several more dayshours, we suspect. This affects a very large number of users, so we are investigating some urgent measures such as moving everyone over to www3, but in a way that does not require you to change the HTML on your page (by redirecting traffic accordingly)... we'll let you know as soon as we have more news. We're really extremely sorry about this outage, and appreciate your sticking with us through this difficult period.]

[UPDATE 3, 16-March-2015 00:15 GMT: The disk array for this server (www4) has now been completely replaced and rebuilt, and is now running reliably again. Maps are working fully and counting properly, but you may still see the latest date for the statistics as '13th March 2015': this will be brought up to date with the latest data over the next few days. Please stay tuned.... ]

[UPDATE 2, 14-March-2015 01:00 GMT: New problems, it seems there are still some 'up and down' issues, so we will have to initiate some deeper investigations regarding the underlying causes. Please stay tuned.... ]

[UPDATE 1, 12-March-2015 11:30 GMT: This has now been (temporarily) fixed, the server is up and running, large maps on www4 will show 'Statistics updated 11-Mar-2015 (and will catch up successively overnight tonight).]

[ORIGINAL POST BELOW]
If you are experiencing unusual delays with ‘Statistics updated’ (not changed since 7th March), ‘recent visits’ or even ‘disappearing thumbnail maps’, and your map’s HTML code begins with www4.clustrmaps.com, please note the following: multiple ‘outage’ periods on www4.clustrmaps.com from 9th March – 12th March (5 hours on 9th March, much shorter periods on successive days) also upset the count refresh, so for some users it means the ’statistics updated’ appears to be stuck and the ‘recent visitors’ (yellow dots) had to restart from scratch.

The good news is that our log file is continuously logging all traffic*, so your count will be updated as soon as we can get to the bottom of the current logjam (which, for the technically-minded is due to a database synchronization issue across our multiple servers). We will be posting updates right here on this posting, so please check back from time to time to see what the current situation. Many thanks for your patience.
——*logging of traffic is only possible when your thumbnail map is displayed

Surprised by a missing map? Read this!

Our free users will normally see ads, courtesy of Google AdSense, displayed in a few places near the large map that is shown when someone clicks on the small thumbnail map. However, there are a few exceptions to this. One case in particular has resulted from an issue that has recently been brought to our attention by the team at Google AdSense: in particular, if a user has registered a URL with us that includes any personally-identifiable information (such as an email address), that information appears as part of the overall URL string you’ll see in the browser address bar and might inadvertently expose that information back to Google AdSense as it serves up the ads. It would look something like this:
http://www.clustrmaps.com/counter/maps.php?url=x@y.z(in other words, the x y and z together look they might include an email address, either accidentally or deliberately)

We are therefore obliged to take steps to prevent such personally identifiable information being shown and being passed back accidentally to Google: one such step is to ‘intercept’ displays of that nature, and ‘redirect’ them to this very page, which may very well be why you are looking at it now!

If you are the owner of such a site, don’t worry: all of your statistics and dots are still stored on our servers, as you can see from the thumbnail map itself.

To see the actual numbers, you can either log in to your ‘Admin’ page athttp://www.clustrmaps.com/admin/action.php
(specifying your full registered URL and password)
or alternatively you can get in touch with us by email to discuss how to migrate to a URL that is more representative of your website (and does not ‘look like’ it might contain an email address).

A few weeks ago we had a user whose thumbnail map had disappeared, but only when viewing via the Chrome browser. She also found the solution, and for this we have given her a free upgrade to ClustrMaps+ and linked to her solution below:

Great news! Remember yesterday the ClustrMap map wasn’t showing up? I fixed it. This morning, I reset my browser settings in Chrome, and voilà ~ it now shows up!

If you want to mention it in your blog, you may want to reference the Chrome page with info (below) on Resetting.

Thanks very much for your hand in resolving the trouble ~ I really like your beautiful ClustrMap map!

Deborah also points out that if you use ad blockers, you need to include exceptions to allow clustrmaps.com

Thanks again, Deborah. If you have any useful suggestions of your own, just email us or Tweet @clustrmaps or add a Facebook wall posting to www.facebook.com/ClustrMaps. Any suggestions that we publish will earn a free upgrade to ClustrMaps+ (better maps, no ads).

Recently, most wordpress.com blogs have experienced a change so that the links to ClustrMaps thumbnail maps and large maps have automatically been changed from http to https. However, this ‘breaks’ the setup we are now using, so your map will not display properly. If you View Page Source on your page you’ll notice that the embedded HTML code for ClustrMaps does not match what we provided you with!

The solution is as follows:

1. Login afresh to your ClustrMaps ‘Admin’ page at
http://clustrmaps.com/admin/action.php
using exactly the URL you registered with.
(If you forgot your password there is an option there to get a fresh one re-sent)

2. Where you see the “HTML (thumbnail map) code for your page”, choose option [B] (minimalist). This will provide new code that avoids getting automatically re-converted by WordPress.com from http to https and will work fine.

3. Copy that code from option [B] and paste it into your blog, i.e. over-writing what you have there now for the ClustrMaps widget.

That will fix it for you. Get in touch if you still experience problems. This is unique to wordpress.com users (for wordpress.org code, users are normally hosting their own blogs, or using a third-party provider, in which case our code option [A], the Javascript variant, can be used, which provides much more frequent map updates.

We are truly humbled and honoured by the great level of support and encouragement we received from so many users. Messages like “keep up the good work”, “we are wishing you all the best”, “we’ve been there, and know how difficult it is”, “you guys are the BEST” and so on: these are the things that keeps us going and wanting to offer the service that so many of you still appreciate. THANK YOU ALL!

SUMMARY: A major disk crash affecting maps on www3.clustrmaps.com has now been solved; a very lengthy recovery procedure occurred using reliable backups. The original post and updates below are in chronological order, so the latest updates are at the bottom. Your visitor counting continues to work correctly as long as your thumbnail map is installed, so your long-term totals will be fine, but data between 29/Oct/2014 and 02/Nov/2014 was not collected at all during the time the actual server was down. See below for the full story.

[Original post]: We received an alert at 10:15 GMT 1st November 2014 that one of our servers (www3.clustrmaps.com) was not responding. The means that anyone who has a map on that server (you can tell because the link to your big map begins with www3….) will be experiencing a data outage during this period [no visitor counting], for which we apologise. Our hosting provider (SoftLayer.com) is ‘on the case’, and it looks like there is a problem with the RAID disk array that requires a disk swap and rebuild. This normally takes a good few hours, but additionally it has taken some extra time to locate the fault. Please return to (and refresh) this blog posting for updates. Many thanks for your patience and understanding.
-The ClustrMaps Team

[UPDATE 1, 15:30 GMT, 2nd November 2014]: Sorry folks, this has taken much longer than we anticipated. The double RAID array had BOTH disks fail. They have now been replaced and restored to a clean state, and we are in the process of retrieving the backup data and reinstalling everything on www3. Stay tuned for further updates.

[UPDATE 2, 21:30 GMT, 2nd November 2014]: Better news – we are still on the case and have retrieved the off-site backups – it was extremely bad luck having both disks fail, but of course that’s what backups are for, so we’re OK. Restoration requires an entire fresh server software installation, which is actually more complicated than it looks, so that’s what is happening tonight.

[UPDATE 3, 23:45 GMT, 2nd November 2014]: The main server code has been restored, but the database and hundreds of thousands of maps (just for server www3!) are being separately restored from backups. In the meantime, you may see incorrect or misleading error messages on big map pages like the following:

This is just a temporary artifact, and if you had a map before, it will be restored, and you are definitely still a registered user! Restoration is continuing overnight and we’ll be on the case again Monday 3rd November.

[UPDATE 4, 13:30 GMT, 3nd November 2014]: Sorry folks, a few factors out of our control are making this a bit of a brain-teaser. We’re still on the case…

[UPDATE 5, 21:00 GMT, 3nd November 2014]: At last we’re in a position to make a few predictions about full service restoration for www3.clustrmaps.com . The main server, disks, infrastructure and user database have all been restored fully, but there is now the detail of restoring all the statistics (the ‘dots’, essentially, plus numerical details) for each of the very large number of users on that particular server. Right now, the restoration is running non-stop, but it could be another two days before that is finished and properly tested. So our optimistic prediction is for the 22:00GMT on 5th November. Our pessimistic prediction, based on how long it has taken so far, is another two days beyond that. If we can do better, we will post that information here immediately. This is a much bigger job than we had thought, and we’re grateful for the messages of support and understanding that we have received from our fantastic user community – thank you all!

[UPDATE 6, 19:00 GMT, 4th November 2014]: Well, we have finally been able to see thousands and thousands of maps on the restored server 3, so things are actually progressing nicely, although at a frustrating snail’s pace. We don’t want to ‘unblock’ those maps until we are sure that they are all visible including thumbnail maps, map history/archives and numerous other fiddly details. Still hoping for a re-launch of the full server 3 tomorrow night!

[UPDATE 7, 10:18 GMT, 5th November 2014]:Real progress now: everyone’s thumbnail maps have been restored. We can see everyone’s full maps on the restored server 3, but now need to update all the counts before we ‘unblock’ those maps: two or three days of visitor counts will be completely missing from www3 maps, corresponding to when the server was down on 1st and 2nd November. Still hoping for a re-launch of the full maps for everyone tonight (optimistic scenario) or by 7th November (pessimistic scenario).

[UPDATE 8, 12:30 GMT, 6th November 2014]: Things are progressing, and you may be wondering why it is taking so long: well, consider that the transfer of the backup files took a staggering 3 days, and that’s over a super-high-speed internal network at our hosting company! There are literally millions of files involved (we’ve got a lot of users on each server, and they have lots of maps, map archives, long data history, etc). So users now have their thumbnail maps in place, and the thumbnail maps are enabling visitor counting the whole time they are visible. But the big maps are only visible to us at the moment, while we painstakingly restore all the processing jobs: there are hundreds of these, and they need individual checking – this could normally be done in a day or two, but the key members of staff have moved, and although they are assisting us, this can only happen on a ‘piecemeal’ basis, so it’s going to take some time: we’re still a few days away, as far as we can tell.

[UPDATE 9, 09:00 GMT, 8th November 2014]: Slowwwwwwwwwwwwww progress… sorry folks; could be as much as a further week now . Harder than we thought…

[UPDATE 10, 22:30 GMT, 9th November 2014]: No real news… a few more hurdles cleared (some of the overnight statistics jobs), a few dozen more to go before we can really release the maps for public viewing.

[UPDATE 11, 15:00 GMT, 11th November 2014]: Fortunately, even more hurdles were cleared last night, and quite smoothly too. Our main problem for the past week has been that even though the data and maps were all nicely backed up, we have been having to reconstruct a large number of the actual map-processing scheduling jobs which have subtle inter-dependencies that can not simply be copied from the other servers. In the future, we’ll be well-prepared for such an eventuality, which has not happened in our 9 years of serving up over 15 billion maps. Sunday 16th November is our current ‘make or break’ day when we’ll know precisely how close we are to a complete reconstruction.

[UPDATE 12, 10:20 GMT, 12th November 2014]: Logfile analysis and map updates were completed successfully on a ‘representative sample’ of maps last night, so now we are in good shape to run these processing jobs on the entire collection of maps on server3 for all the data collected over the past 10 days. This is an extremely big task, so we will break it down into chunks, so this will take a few more days – the next update will say how it’s going.

[UPDATE 13, 21:50 GMT, 13th November 2014]: We are now reprocessing the data going back to our logfiles on server3 which were stored on 02/Nov/2014 at 22:02:07 GMT, which is great news, i.e. your visitor counts for maps on www3.clustrmaps.com have been running correctly since then, as long your thumbnail map has remained in place. A few days of counting were lost in the interval between the last saved log files and 02/Nov/2014 at 22:02:07 GMT, and we are now hoping this will be a relatively short gap. The worst case will be a gap from 29/Oct/2014 to 02/Nov/2014, but it may be shorter than this: we will report back as soon as we know.

[UPDATE 14, 11:15 GMT, 16th November 2014]:Great news. We have just completed a conference call with the key developers and gone through every step, and have been able to confirm that every single cron (batch) job has completed successfuly for several consecutive nights, so we have now released the full maps with latest updates to everyone. You do not need to make any changes to your code. There will be a small ‘gap’ in the dot processing between 29/Oct/2014 and 02/Nov/2014, but we think this is a small price to pay for being able to regain all other data during a rather painful recovery process. As long as your thumbnail map has been in place, visitor counting for maps on www3.clustrmaps.com has been continuous since 02/Nov/2014. Many thanks again to everyone for your patience and overwhelming support — we are extremely grateful for all the wonderful supporting emails we received from you!

[FULL SERVICE IS NOW RESTORED, AND UPDATE 14 ABOVE IS THE END OF UPDATES TO THIS BLOG POSTING]

We have disabled ‘pre-order signups’ for ClustrMaps 2 until further notice. Please check back here on our main blog page for updates and availability, particularly concerning the re-opening of registration for the free ClustrMaps service (NOW AVAILABLE, as described here).

[UPDATE: the problems mentioned below have now been fixed. Server 4 ('www4.clustrmaps.com') was restored to full service at 02:12 GMT/UTC on 20th May 2014, a total outage time of 28 hours and 6 minutes. See 'implications' at the very end of the post.]

We were alerted to an outage of one of our servers (’www4′, i.e. affecting any maps beginning with www4.clustrmaps.com….) at 23:06 UK time last night (22:06 GMT/UTC on 18th May 2014). Our hosting provider, SoftLayer, is undertaking a detailed investigation, but there appear to be a number of more complex issues that are taking some time. Normally, short of outright hardware failure, these fixes should be done within a few hours. In any event, please stay tuned for updates here.

In all likelihood this outage will have affected our overnight map updates for one or two days.* We will post tomorrow to keep everyone fully in the picture.

We are deeply sorry about this occurrence. Any premium (ClustrMaps+) users badly affected by this will naturally be entitled to a pro rata refund, or full no-questions-asked refund if they prefer. Thanks to all of you for your patience and understanding!

*[Implications: for those with maps on server 4, your last map update will have taken place on 18th May, and would have resumed on 20th May, meaning that, depending on where your map sits in the processing queue, one or two days of 'visitor dots' will simply not have been collected. Also, you will see 'Statistics updated 18 May 2014' for an additional day (until the next overnight processing occurs), and the 'recent' visitor list beneath your large world map may take up to an additional day to start appearing again. In the long run, you will barely notice the 'dent' in your grand totals, but please contact us by email [see address here] if you want to start afresh, or if you are a paying customer interested in a partial or full refund, are otherwise concerned. Thanks again to everyone for your messages of support.]