ManageWP Orion Developer Diary #3: Bulletproof Backup Solution

Update: As Orion development evolves, some of the information below became outdated. Check out the Dev Diary #8 for more up-to-date info!

A lot of you have patiently waited for us to deliver on our promise and provide you with a bulletproof backup solution that your websites direly need. It’s time to reward your patience – the new backup app is being released tomorrow, on May 5, into the ManageWP Orion closed beta and it’s jam-packed with new features.

No More High Server Load

The first thing you’ll notice is how easily your website server is running backups. The new Orion backup runs just a small fraction of the process on your website server; the vast majority of the work is done by our own server infrastructure, built specifically for this purpose. With the help of our Hibernation technology, your website is queried for content, which is then uploaded to our Hibernation Pod (our cloud storage infrastructure) where a backup archive is assembled. And the best part? The load on your site server is little more than a regular website visit.

Real-time Protection

Scheduled backups are obsolete now that we’ve switched to real-time protection. The new Hibernation tech allows you to run incremental backups every hour, 24 hours a day, 7 days a week. That’s 720 restore points each month!

Compatible With Huge Websites

With the resource issues resolved, there is no upper limit to the size of the website you can back up, both in terms of the physical size and the number of files. The backup is incremental, meaning that after the initial backup, only the changes you make are tracked and uploaded. Instead of creating a 10GB backup each and every time, you just run a backup for those 2MB for your latest post.

Secure Off-Site Storage With Hibernation Pod

Storing backups on the website server is a bad choice for a number of reasons – hackers could access your FTP and hijack the whole website; with larger websites you could easily burn through your physical space; but most importantly, if something happens to your server, your only chance of getting things back on track has crashed with the server. That’s why we’ve decided to assemble and store your backups on Hibernation Pod – our cloud storage solution that makes it ridiculously easy to store and restore your website backups.

We Play Nice With Everyone

Some managed hosting companies, such as WP Engine and Pagely, restrict PHP backups because they interfere with server performance. This is no longer the case. The traffic and server load are only a little higher than a regular site visit, so there’s no need for the hosting companies to throttle or terminate the ManageWP Orion backup script. This is very important because you ask us all the time if ManageWP plays nice with hosting company X – from now on we play nice with everyone.

Why We Chose to Build the New Backup

You know how the old ManageWP backup works. It’s a PHP script relying almost exclusively on your website server resources, similar to other major backup plugins such as BackupBuddy or Updraft Plus. It’s quick, it’s dirty and it’s resource intensive. The problem is, if you run a serious business, anything less than a 99% backup success rate is unacceptable. Our research showed that ManageWP and similar major solutions deliver ~80% success rate with default settings, and in some cases require a lot of compromises to run reliably. You probably know all this all too well.

We explored all possible options and concluded that further optimization would hit the ceiling at ~85%. That’s why we came up with the Hibernation technology. It allows us to achieve that magical 99% success rate by running low-intensity processes off-site and utilizing incremental backups, Hibernation Pod storage and a number of other optimizations.

Getting Started With ManageWP Orion Backups

First off, say goodbye to complicated scheduled backup forms, credentials and account linking. We’ve made the user experience as straightforward as possible. When you log into your ManageWP dashboard, you will immediately see a rundown of your websites in the backups widget:

If you’re a ManageWP user, the first thing you’ll notice is that the scheduled backup tasks are gone – with real-time backups there just wasn’t any need for the hassle of juggling multiple backup tasks; now all of your websites are sorted into categories, according to the status of the backups:

Automatic backups: The backup for these websites works like clockwork, yay! You have the option of running a manual backup or going to a detailed backup report for that website

Backups not enabled: For those rare occasions when you don’t need a backup for a particular website. You can always check this category to enable the backup

Clicking on the detailed backup report gives you an in-depth backup report for that website:

In the upper left corner of the screen you’ll see the backup calendar with information about your backups. There are two buttons: Backup Now if you want to create a backup right away, and ON/OFF, which is the only button you need to click to turn your real-time backup on and off.

The lower left corner lists the available backups for a chosen date. Clicking on one of them brings up a detailed report on the right of the screen:

The lower right part of the screen is a screenshot of your website at the moment the backup was created – this is great if your theme broke down or the website has been defaced and you’re looking for the latest functional backup to restore. Above the screenshot, you’ll find more information about the backup: WordPress version, the number of plugins, posts, comments, the latest published post and the active theme.

And last but not least, the upper right corner is where the magic happens. The Download backup button allows you to store the backup archive on your local computer. Test restore backup is great for critical websites – your backup is restored on our test server. You can see what it looks like and once you’re certain that everything is in order, click on Restore backup. This button restores your website using the backup from our Life Pod storage – a simple single click to start the restore. Enjoy it as the progress bar races to 100%

Progress bar – a developer’s best friend

Please note that at the time of writing we’ve disabled the Test restore backup and Download backup buttons – once the internal tests are successful you will be able to play with them to your heart’s content.

That’s it for this blog post. We have a lot more going on, so expect the next Dev Diary next week, with more information about Orion apps. If you have any comments, suggestions or feedback, please let us know. We’re very excited with the road ahead and the tools we’re building to help you run your business efficiently and reliably.

Nemanja Aleksic

Karlos

This looks fantastic, have tried quite a few different options to try and get around the resource issue. Some sites just will not reliably backup and it puts a huge drain on our time having to ensure that backups are complete and completed properly. It would be awesome to start testing out this platform, would love an invite if you are still handing them out.

Dirk-Jan

natebald

I have expressed my opinion in a support ticket, but will do here as well. It think the backup solution goal is incredible, but without offsite backup locations it is far from a successful product.

What if your service OR Amazon (PODS) is not available? If it is on a NAS on my network or even dropbox, it is always available AND secure. Also, I like to keep 8 days of running DB backups, 5 full site weekly backups and 3 full site monthly backups. This is standard disaster recovery 101, If I get a virus say, I can restore for a backup 3 months prior.

If your service is not available, say the internet goes down, I have a copy here on my network and my developers can continue to work.

I can’t stress this enough, remote backups are invaluable, if you haven’t got much feedback yet stating that is is because people haven’t realized it yet.

Nemanja Aleksic

Hi Nate,

These are all valid remarks, but there are three reasons we chose to build the backup the way we did:

1) We cannot improve the reliability of the backup if it’s not going to the destination we’ve configured for this purpose
2) Storing files on another external storage carries additional expenses and effort; there are other tools that have a much higher priority right now
3) A recent survey by CodeGuard showed that only 24% percent actually use backup plugins, even though the vast majority thinks backups are important. The hard truth is that even the WordPress professionals want as little hassle as possible with the backups, which is why we cut down the ~12 decisions from our current backup interface to just one (On/Off) in the Orion release.

roller

Hi
Really excited to try the new backup! I currently have 2-3 sites that just won’t do to on your current system. I’m near launching a new WP maintenance addition to my business and will really need the backup. I was informed on June 8 that I would get an invite but so far have not? Any other steps I need to take?

Nemanja Aleksic

richard

This sounds really good as have had some issues with automated backups recently to my present hosts. Would love to try this out as an alternative to Dropbox although I have full pro account there.
It will be good to see what the costs are to have such a reliable backup service.
Well done guys.
Please add me to the next release.

Nemanja Aleksic

hello

This looks incredible guys, amazing job! I am just starting out and trying to automate many tasks across numerous sites, such as backups and plugin updates, and would love to help test the beta if possible!

hello

Hi, this looks like an amazing addition to ManageWP. Would I please be able to join the beta to help testing? Just starting out and trying to find an ideal solution for certain tasks across multiple websites, and this looks pretty close!

graham

I personally also am holding my breath in anticipation for this update. My biggest issue at the moment is that backups keep failing and cloning often doesn’t work either. If you would like to add me to the beta – I would be most grateful

Ricky Onsman

I am already hooked on Orion. I now have all 42 of my own and client sites on the beta, nicely sorted into 6 groups, all backing up smoothly. The way they are organised, the speed of the response on actions I can currently perform like spam comment management and plugin updates, the display of things like Analytics … the whole package is efficient, stylish, easy to use and thoroughly seductive. There is more to come, I know (and backup to Dropbox is up there with what I’d like to see) but it is clear MWP is setting new standards with Orion. I have previously been part of the 2.something % of users who drew on support for various issues but it looks like that is in the past. I look forward to your continuing progress. Cheers.

Piet

The new backup feature is really great and works so smooth.

At first I didn’t quite understand why my site would need backing up every hour, but I think it is easier to implement a “one-schedule-fits-all” like this, than to offer all kinds of different schedules per account/site.

I was interested to know how well the restore function worked (something I think people usually don’t want to use) so I destroyed my test site after running the first few backups.

Then I clicked the restore button and BAM! Got my whole site back, just like the way it was before I destroyed it 😀

Nemanja Aleksic

Just the WordPress tables.
If we ran backups of the whole database we would face too many unknown factors and too many MacGyver scenarios – each one of them could cause an avalanche of issues down the line.

jjsararas

Good morning folks! I just happen to have been swamped for the last 3 days and have finally just now had a chance to go play with the new backups. I guess I haven’t been paying close enough attention, but I didn’t realize it was going to be incremental – this is fantastic! I love the UI choices you’ve made too. It’s all making sense and I’m very excited to dig into it a little more. I’m sure I’ll find something to request 😉 Thanks for your dedication to quality!

tony.catone.seo

karl_s

Impressive. Can’t wait to see it in action (or NOT see it, if you know what I mean). Just another reason to choose ManageWP over the competition, and another reason ManageWP is superior. Keep up the great work, folks!

Peter

Jane

This sounds great, would love to be notified when I can signup to use this Orion backup service. I think this service will just what people need for a backup service. At present get backup up from another company but I would love to try this out to see if its better. Many thanks

Nemanja Aleksic

Ricky Onsman

Alright! I’m one of those for whom my 40+ client site backups have been a right royal pain, even with ManageWP. So I trialled Orion’s backup facilities with four of my most problematic sites: one that is self-hosted by a client (has never backed up at all), one that is hosted by an external host (sometimes backs up, mostly doesn’t), one that I host but for some reason has never backed up via MWP) and one that I host that backs up well via MWP but is dauntingly large (10 years of posts will do that).

First attempt at using Orion’s backup: complete success for all four.

Check this morning to see if incremental backups are working: complete success for all four.

For all of you who think this feature sounds amazing, fantastic, awesome: you are 100% right.

Congratulations, Nemanja and ManageWP, you have completely changed the game for multi-site WP backups. Stunning work.

I can hardly wait for all of Orion’s Apps and Preferences to be up and running.

Ricky Onsman

portmaconline

Hi Ricky,
Thanks for the vote of confidence! I am another one who has had more than my fair share of problems with backups, as Nemanja can tell you. Most of the time they have been server-based, and I have had to get my web hosts to increase setting after setting to get even a reasonably reliable backup regime working for our clients, so I for one will welcome this upgrade with open arms!

I might even fly from Australia up to Russia to buy the team a beer or five.
Terry

Nemanja Aleksic

Terry and Ricky,

Thank you both for working with us on a solution and for your seemingly endless patience. Your feedback helped us on several levels, and I’m ecstatic that you’re finally happy with the Orion’s progress – it’s exactly what keeps us pushing for an even better, more reliable and innovative solutions.

BTW Terry, we’re not Russian; Majority of our team is from Serbia, a small but tenacious European country:http://bit.ly/1RfaeqZ

Nemanja Aleksic

Yeah, “multiple WP sites” makes much more sense – didn’t have my morning cup of coffee, so I was all like “When did we finish the multisite integration? Actually, when did we start the multisite integration?”

Adam Hansen

That sounds both good and bad. I like the idea of being able to mix and match, but I worry about how it will end up costing in the end. I am currently on a lifetime 30% discount with ManageWP from taking advantage of a very early promotion. Will that still be honored with the Orion update?

mjmurphy53711

Nemanja Aleksic

Nope. Committing to a 99% reliable backup means we need to control the whole process. There are several technical reasons behind the decision of dropping Dropbox for now, we’ll go into more details in one of the next dev diaries.

Debayne

Nemanja Aleksic

The dashboard itself is fully functional (website overview, plugin and theme updates, analytics, etc.). The client reports and backup & restore have also been implemented. Full functionality should be achieved in the next month or two, when we roll out the public beta.

Nemanja Aleksic

john.cope

craig

I have a particularly awkward site that is over 2Gb and have had to cease backing it up – I think my web hosting company aborts the process. Is there any chance of getting in on the beta to see if it works with this site??

thom

Nemanja Aleksic

mel

So excited for this feature! Been waiting impatiently!

1. When will we be able to download, send offsite or restore backups (any will do)? It’s essentially useless for now and an issue given the backups don’t work on some of my sites in the regular version. When can we see these features functional?!

2. Will we be able to exclude folders, like in case we use another backup like BackupBuddy, so would be good to ignore large unnecessary files.

Nemanja Aleksic

Hey Mel!
1. You can already restore backups, and they are being stored offsite in Hibernation Pod (our cloud storage). Download and Test Restore are coming soon.
2. Other backups like BackupBuddy are already ignored automatically – this was our number one priority since in our current version of backup we had users with up to 20GB of various backup archives just lying around on their websites. We’re still discussing the most efficient way to handle the include/exclude issue. One way would be to handle it automatically; my personal favourite is to give you a tree view of your website so you could click your way to an ideal backup. Will keep you posted!

Nemanja Aleksic

The backups are assembled and stored offsite, on our own cloud infrastructure called Hibernation Pod. From there you’ll be able to download the backups if you wish and store them anywhere you’d like.
The multisite is currently a hotly debated topic – if we will introduce multisite support, we need to greatly improve compatibility over what you have in the current ManageWP version.

billy

Thats great news, I like ManageWP, but am forever getting failed backups on larger sites which made me think of moving to Vaultpress. I am glad to hear of the changes and will try them out when I get a chance. Good stuff.

ellegaard

jldesign

scott

Just a general Kudos for taking a great project and making it stellar. Well done. Then… one wishlist item:
It would be great if you could improve the client reporting feature with a) ability to write a short note to our clients; b) a more detailed reporting/dashboard style (like Google analytics) showing what we do for them each month.

Nemanja Aleksic

Thanks!
I’m happy to inform you that both items are already planned for the next client report upgrade, but I’m not going to spoil the fun of reading the next developer diary that will focus on the client report and its future upgrades.

Nick Defoe

My impression was that BackupBuddy didn’t work on WPEngine due to the architecture of their system, instead of being specifically disallowed due to resources – is that wrong?

We are definitely interested in giving this a go. We have been relying on WPE’s backup solution and backupbuddy for our other sites in the past. Hopefully this can bring everything under the same solution!

Nemanja Aleksic

WP Engine has a server performance and security commitment to their users that they need to keep, so they disabled a lot of functions in their server configurations, namely proc_open. This forces us to use a fallback PHP script which uses way more server resources and is more more likely to be terminated by the server.

I’ll send you the beta invite within 24h so you could test it out and let us know how it went

Nick Defoe

Huh – that’s interesting. I always thought they had some odd cloud file architecture that prevented proper backups using traditional tools. I was also wishing they would add support to their backup solution to link to an S3 bucket or similar for 3rd party storage, but looks like you guys are beating them to the punch!

webdesign

ARS Cons.

Wow. Just wow. As an ex-hosting provider, versed with all the issues of server load, and CloudLinux’s resource-limiting problems, this is quite a feat. I’m on Orion now, will give this a run, expect all my clients to be added to my account, if this is as amazing as it sounds !

Illustrate Media

This is going to be an incredible boost to my workflow if this works as well as you suggest. How to make a great servic even greater. Any chance I could get in on the beta? I have a particular site that I have in mind to test the backup process with.

Nemanja Aleksic

Karl Sandoval

Sounds like really useful and well thought out options – looking forward to testing these eventually. We’ve run the free-5 limited account for some time now, to test and see if this is a useful solution for us, which I think it is. I’ve been looking forward to upgrading to a “proper” account for some time, and it sounds like Orion will be a whole lot better still. Exciting.

Nemanja Aleksic

I don’t know if you’re aware of this, but a while back we decided that in Orion you’ll be able to add an unlimited number of free websites, as well as be able to mix and match free and premium websites, so you could keep everything in one place.
That’s about as proper as you can get

alexander.degroote

Hi, that looks fabulous.
I’m very interested in the new backup solutions, the current backup solution doesn’t work perfect for a huge site like mine and I waite to join my other sites. Is it possible to join the bèta te help testing the new backup solution ?