Today, after nearly 4 months of development and refinement, Backup Pro 3.3 has FINALLY been deemed stable enough for release. It's been a LONG time coming, but the wait has certainly be worth it. This release includes your normal round of bug fixes, a new CMS implementation, and a really cool feature.

First, Backup Pro 3.3 fixes a couple bugs related to the file system oversight. A few users had submitted issues about their backup limits not being implemented and/or respected when backups were ran from the command line. So that's fixed.

Next, this release is the inaugural release for Backup Pro for Concrete5 (CMS Critic's Best CMS for Designers award of 2015). It's available in the Concrete5 Marketplace, ready for sale, and has been approved by the PRB (the special group that oversees packages). In fact, thanks to them, Backup Pro has quite a bit more finesse than it did, thanks to issues they found (that have been resolved) around SFTP file system handling and relative paths. So they made Backup Pro better, essentially.

Last, Backup Pro 3.3 adds a new feature that's probably not that interesting to most people right off the bat; a REST API Server. But it's a necessity to allow for this: the Backup Pro Manager. Expect a followup post on the Backup Pro Manager, but suffice to say, it'll allow you to control all your Backup Pro installations remotely. But that's another topic for another day.

The REST API Server, though, allows for complete control. Everything you can do in Backup Pro can be done with the REST API Server. Everything. Plus, authentication is handled using HMAC (just like Amazon S3 FWIW), so it's secure, it's an optional feature that's off by default, and there are quite a few endpoints to get work done. There's even a client library all ready to go to work with on Packagist to make life easier if you want to work with Backup Pro programmatically.

About an hour ago, after working 14 hours straight mind you, I was able to run the Concrete5 version of Backup Pro through the entire BUILD process and have all the tests and release archive come back good. No failures, no issues, and it, at least for now, appears to have no real issues. At least insofar as the standard Backup Pro requirements. So a small public release is called for to let real users kick the tires and make sure it's up to Concrete5's standards.

I plan on Backup Pro for Concrete5 being available for free, though unsupported, download at least until it's included into the Concrete5 Marketplace and made available for sale. But even that's a good while away from even starting. And, to help incentivize the help and testing, anyone who participates in the beta will get a free license once it's finally officially made public.

Know that Backup Pro for Concrete5 requires at least 5.7 and all the normal system requirements of Backup Pro so it WON'T work on versions less than that. Other than that, go nuts; find a bug, report it in the comments. (If this turns into more than a couple people I'll move it into something more official but for now, let's keep it informal.)

So last week's Magento run was a bust. Went a whole 4 days this time before the pain got the better of me. But, hell, I figure since I'm the one who decides what to do I might as well NOT do painful things. So, as mentioned, Magento is on hold. Which means I'm free to do Backup Pro for Concrete5 instead. Which I have been since last Friday.

And it's actually gone really well! REALLY well. Who woulda known Concrete5 had some game in the API department? Painless. Truly. Yes, it's well documented and there are quite a few examples available to learn from, but more importantly the code itself, while not having a single docblock in the bunch, is as legible as a coloring book. Just zero clever for the sake of clever; it's a very intuitive codebase.

In fact, Backup Pro for Concrete5 is so painless to put together that it's almost done. Well, almost done enough for a public developer preview by tonight at least. There'll likely be some things not working right, as with every developer preview, but it'll be feature complete. All I have to do is finish today before release is database backup restore, Cron URL abstraction (for automation), setup the Integrity Agent logic, write the Build script, and a couple rounds of QA. I'm hoping to have it passing all the tests and be a perfect build BEFORE release but it's possible I may have to manually build the first few developer previews. Big day, but not untenable.

I think, with Backup Pro for Concrete5, I'm going to hold off making it available for sale until I can get it admitted into the Concrete5 marketplace. This way the community can kick the tires for a good while and make sure it's perfect and up to their standards.

If you want to participate in the developer preview be sure to follow mithra62 on Twitter or the mithra62 Facebook page. I tend to just post links to download the releases there so it's all informal like.

Let's not bury the lead; Backup Pro for Magento is on hold. For now anyway. If you've ever done any Magento development you likely know why, but I'll go into detail below for those who don't. I want to be clear though; I'm not saying it won't happen, it HAS to happen at some point, it's just a "not right now" sorta thing. Instead, and as I have a history of being capricious this may change, I'm doing Backup Pro for Concrete5 instead.

Magento's a weird beast. It's extremely popular and powerful, certainly, but from a design perspective it's just impractical for the purpose. Put another way, Magento takes itself VERY seriously. To a fault. A very very painful fault...

Which is the crux. Magento is painful to develop for. In nearly every way, Magento appears to go out of its way to get in yours.

For example, Magento has an MVC design. Sort of. It just had to invent its own interpretation of it is all. So if you're used to the whole, "have a controller and pass data to a view script" paradigm, you're in for a shock. Blocks, Grids, XML, and Helpers are your life and it's just completely contrary to most other platforms. But, and this is important, it's the embodiment of a painful platform.

This is now the second time I've looked at the mountain of Backup Pro for Magento and went, "meh, screw it". There are TONS of other Platforms out there. Like Concrete5, for example. I'll DEFINITELY be revisiting Backup Pro for Magento in the future, the potential revenue's too compelling to walk away from, but there are lower hanging fruit to go after that don't have the mental overhead of Magento.

And I've decided on Concrete5 for the next Platform for a Backup Pro version. Concrete5 is a REALLY compelling CMS. Elegant actually. It has an actual MVC pattern that makes sense and doesn't require mental gymnastics for the simplest thing. Seems to just work.

I'm still in the "kicking the tires" phase but Concrete5 looks promising. An elegant and fancy design. Modular. An actual MVC paradigm. And, most important for my purposes, a marketplace for selling addons, or, as Concrete5 calls them, Packages. So, you know, Backup Pro'll have an immediate audience.

Continuing where yesterday's post left off, Backup Pro for Magento has a long way to go. This morning, after working all day yesterday, things are pretty well barely started. I did make quite a bit of progress yesterday, just not enough that it looks like something was done.

First, the installation stuff's all done. Cross that off the list at least. Like most things, Magento has a unique way of handling things like extension installation and update, at least when compared to other platforms like ExpressionEngine or Craft. There's that whole directory and XML quasi namespacing thing they do. Just complicates the day.

Then I added ACL controls to the Backup Pro administration menu. That, too, was all about XML, but actually pretty straight forward enough. It's really important that Backup Pro, for all Platforms, function as natively as possible using the host Platform standards. And Magento does their whole ACL stuff. But that's done too now.

The big success yesterday was getting the admin controllers working properly. That's just a PITA. Painful. There's a whole paradigm that was really tough to visualize until I started hacking up the Magneto core adding breakpoints. But I finally figured it out. Finished this morning's work with the basic working outline for 3 controllers, so not too shabby.

Now I need to get the layout stuff going today. Magento is killing me with its absurd logic and outdated design. Looks like there isn't a way to, from a controller, setup a view directly. It's all Helper this and Helper that. So that's my day today. Yay...

I'm hoping to have the basics of the Backup Pro Dashboards worked out by morning; so that'll be main Dashboard, and the file and database backup management Dashboard. Should be doable.

Now that Backup Pro for PrestaShop's been out for a couple months, and I've had a chance to implement the first round of feedback, I've restarted development of Backup Pro for Magento. I tried putting it off as long as I could, Magento's a little more painful to develop for than other Platforms, but it's time to suck it up and get it done. To help make it more fun, I thought I'd document the process of creating a new version of Backup Pro starting with day one, which was yesterday actually.

So, yeah, day one. Done. Actually, now that I start writing this I realize how little's been accomplished really. I spent most of the day in research, re-familiarizing myself in just how fuck Magento extensions are put together. They really are unique and just lacking in anything resembling elegance. It's weird knowing it was intentional. Say what you will, but Magento is software that takes itself really seriously. It seems pretty clear this whole extension layout and structure was designed to work this way. Like, on purpose.

So there's not much done. I have the basic file structure setup for the extension. Backup Pro displays in the Magento extension page, the Administration menu is being displayed, and a basic front end controller is setup for testing the Composer libraries work without issue. But there aren't any Administration controllers setup yet. Don't even have the basic structure in place for controllers yet. But it's in process.

I stopped last night while getting the installation stuff worked out but didn't get too far with it. I'm missing something about the code paradigm so a bit more research is needed; should have it worked out soon enough though. Once that's done, I still have to do some testing to ensure the namespacing works right (I'm a little concerned Magento's autoloader could conflict with Backup Pro's default setup) and then it's off to get the Admin controllers rolling.

I just released Backup Pro 3.2.2 which is sort of a hodgepodge of new features, bug fixes, and refinements for all platforms. I'd been working my ass off trying to get it approved for inclusion into the PrestaShop Marketplace so this build's dedicated to them and their undocumented requirement for PDO instead of mysqli for approved products.

First, the big lead is that Backup Pro now allows for either PDO or mysqli for database interaction. If you don't get that it's probably not that important, but it does mark the start of a larger strategy: support for more database's than MySQL. So, PDO was needed regardless of PrestaShop's requirement

Next, you can FINALLY store backups in buckets using dot notation. Previously, if you created a bucket named, for example, "my.bucket.name", Backup Pro wouldn't be able to find it on Amazon S3 so would fail to validate the bucket for use. The problem, for the curious, was that Amazon has different rules for bucket names depending on the bucket region, something which Backup Pro previously didn't let you set. so now you have to select your bucket region, but can also have dots in your bucket names. So there's that. You can thank/blame Brett Burwell for that.

The other Storage Locations got updates too. SFTP had a bug with NOT validating the host-name properly; that's fixed. And Google Cloud Storage and Rackspace Cloud Files can both have sub-folders/prefixes set so buckets can be easily shared amongst installations.

Say you're going to do something insane. Absurd even. But also potentially rewarding beyond anything you've yet achieved, not just financially but also in more ephemeral ways. Something nobody else has done. Unproven. It'd be a fair assessment that things are going to get weird at times. Different.

This is my situation with
Backup Pro going cross platform; there are very good reasons nobody does that. Quality's going to be a nightmare. Customer service overhead has the potential to overwhelm a dev. Concepts like consistency and quality are very real problems. Colloquially, it's a fucking nightmare of a project plan to put it mildly.

So, obviously, early on I knew I had obstacles; I didn't go into this blind by any means. Treating Backup Pro as a traditional project wasn't an option. Concepts like a single and unified code-base (a cornerstone of web development) aren't just impractical but almost impossible. So, again, early on, I knew I'd have to do things differently for Backup Pro. And that's when I realized the solution was going old school.

Asset Management

As web developers, we tend to think of our projects as starting in a single folder. For me, on websites, it's usually the folder above the webroot. That's the center of the versioned files, the IDE root, everything. For example, for mithra62.com, everything lives in a single directory. Even as an add-on developer for ExpressionEngine (and WordPress back in the day); a single folder was where everything lived. Backup Pro, pre 3.0, was stored in a single folder that was symlinked into my dev site accordingly (for theme and third_party folders) and that main folder was stored in a Git repository. Even things like the tests and deployments all centered around that single folder. Traditional.

But Backup Pro can't be a single folder. It just can't. Not anymore. There are 4 unique and individual components involved in every release; the core Backup Pro files, the Jaeger files, the Composer files, and the Platform files (the EE module files) . Symlinks abound to manage that. In point of fact, Backup Pro, as it stands now (for EE 2&3, Craft, WordPress, and now PrestaShop) is made up of 7 individual and stand alone components for all platforms.

Quality

Just due to the nature of programming, a change in Process A can affect Process B which cascades to Process C and Process D. But this is a known problem regardless. That's why we test and debug and test again and debug again. Concepts like unit and behavior testing are huge in getting around these types of issues but with Backup Pro being cross platform this problem's compounded to the nth degree. It's just not practical to personally check every single combination of settings and use cases across all the platforms Backup Pro supports. Not by myself. Not with any consistency.

When Backup Pro was just for ExpressionEngine, it wasn't nearly so difficult to stay on top of things. I'd just notice issues or problems during dev and fix or change or do whatever needed being done. Traditional as well. But now that a change to accommodate WordPress can also affect the Craft version things aren't nearly so neat and tidy.

Consistency

This is the biggie. When you use Backup Pro for ExpressionEngine it should be the same experience as the PrestaShop version as the WordPress version as the Craft version. The same messaging and logic and peccadilloes and, well, as much as conceivably possible should be the same regardless of platform. There can NEVER be anything you can do in one version you can't do in another and how Backup Pro works for one version is ALSO how it MUST work for the others. Otherwise, what's the point?

Consistency's even more complicated by the fact that all the Platform's have vastly different design and creative direction and tools. PrestaShop is built using Bootstrap, ExpressionEngine 2 & 3 have it's own custom front end framework, and on and on (thankfully, they all use jQuery at least). They're all different so each requires a unique design. I have no control there, but the things that make Backup Pro Backup Pro, I do have control over. So, messaging and experience are problematic to put it mildly.

Manual BUILD'ing Just Plain Sucks

So, let's recap. Here are the challenges so far:

Backup Pro, for each Platform, is made of at least 4 unique sets of files and directories all managed individually.

Due to the nature of Backup Pro, the risk of bugs and issues is absurdly high.

Consistency of the experience across all Platforms is crucial. ''

So, as a responsible developer, here's what my release flow would look like to solve everything when I have a new version of Backup Pro:

Take the various files and consolidate into an individual package based on the Platform. EE gets 1, Craft gets 1, and on.

Do a clean installation of the new package and visually verify everything works as expected.

Run through the unit tests

Archive everything up into a zip file for distribution

And I'd have to do that for every Platform. Which, as of right now, is 5 times. For every release. Manually.

No. Thank. You. That would be my life. I'd constantly be in the process of getting releases ready.

Phing to the Rescue

Automating the process was the only viable solution. And that's where a lovely little tool called
Phing comes in. Phing's "a PHP project build system or build tool based on Apache Ant". So it's a little tool that'll allow me to do things autonomously. Cool. So I did.

Here's the current ExpressionEngine 3 BUILD XML I use to put a Backup Pro release together:

What that magical little file does is control the entire process of taking all the components that make up Backup Pro and convert it, autonomously, into a viable product ready for purchase. Now, instead of manually doing everything, Phing just reads in that file and does it all for me. I'm getting REALLY good at fencing (the chair's a problem still though but I'm working on it).

All told, the process takes around 20 minutes per Platform but it runs nightly (on a schedule) and makes sure Backup Pro is ALWAYS stable and ready for release at a moment's notice. Here's what it does:

Deletes everything in the existing build directory so there isn't anything left to compromise things

Checks out the current, working branch, for all the Platform scaffolding (EE module files, for example)

Copies everything to it's Platform specific destination

Checks out the current, working branch, for all the mithra62 libraries needed and copies where needed

Copies the Composer files over (removing anything unneeded like tests and example files)

Moves everything into a clean installation of the Platform (nothing else installed or configured)

Runs through the unit tests (500 tests and 2200 assertions to date)

Runs through browser based, Selenium, tests where it'll install the add-on into the Platform, check something, then uninstall the add-on

Compress the file and store in the release archive directory

To help visualize this I've created a video of the ExpressionEngine 3 BUILD process.

Today's release of Backup Pro 3.2.1 is a maintenance release that includes mostly bug fixes though there is an update to the Database Restore pages so they just work nicer. It's not strictly required that you update unless you're experiencing any issues or just want the latest and greatest. Be sure to take a look at the Change Log for complete details.

Earlier this month, before the holiday, I quietly released an update to Backup Pro that crossed 3 major items off my todo list; SFTP and Dropbox Storage Engines, for storing backups on said locations, and Console Routing, for taking backups from the command line. Those are 3 of the most requested features for Backup Pro, since the first release actually, so having them finally completed is pretty big and should help ease the pain of your backup. Oh, and on top of that, Backup Pro 3 is also where the PrestaShop version is available for sale and download. So, yup, pretty big release.

SFTP and Dropbox Storage have probably been the 2 highest requested features for Backup Pro since it's release waaaaaayyyyy back in 2011. And, duh. Dropbox gives 2 gigabytes of storage for free and SFTP is just better than FTP (it's not even worth discussing it's so obvious). And, since Backup Pro allows for multiple storage locations for your backups, why not use both? Be sure to check out the docs for details on setting those up.

Console Routing though... Console Routing's where I give a big middle finger to PHP and it's dumb ass decision to use a configuration file. (Did you know PHP is the only programming language with a configuration file)? The configuration file's where things like memory limits and execution time limits are set. But they're NOT when PHP is used from the Console. So, for example, if you have a big database on a conservatively configured server Backup Pro's STILL got your back. It'll just work. Of course, taking backups through the web interface will still be a problem, but automating backups (or manually taking backups trough Console Routing) won't be a problem at all.

On top of aaaallllll that, Backup Pro 3.2 is also the official release of Backup Pro for PrestaShop. PrestaShop is one of the top e-commerce systems available and now has Backup Pro available for it. I'm currently trying to get Backup Pro into the official PrestaShop Marketplace, so you can purchase Backup Pro straight from within PrestaShop, and will update once that's complete. For now though, if you want to get a jump on things, you can purchase Backup Pro directly from right here.