The high level explanation is that Veeam is using both TLS and SSLv3 components to connect to VMware – and VMware has disabled the SSL method in v6.0U1. There is a bug in how Veeam is auto-detecting SSL or TLS connectivity, causing this issue. Other VMware products are having similar issues talking to their own products, from what I understand.

Veeam has a KB2063 on the issue here: http://www.veeam.com/kb2063 You have two options – call in and request a private out of band hotfix from Veeam, or make changes on the VMware side.

I’ve whipped up a quick BASH script that seems to work in my testing. It will:

· see if the desired option exists and exits

· if the options exists, but is the opposite setting (true vs false, etc) it will flip the setting

· if the option does not exist, it will add it

TEST IT BEFORE YOU RUN IT IN YOUR ENVIRONMENT, I’m not responsible if it does wonky things to you.

Applying the changes does NOT require Maintenance Mode on the hosts, or any Veeam service restarting. You can simply “retry” the job on the Veeam server, and “It Just Works”

This will likely be resolved by end of September when Veeam releases the next update to Veeam B&R – or there may be a vSphere v6.0U1a released. Once the Veeam fix is released, it may be prudent to reverse or disable these changes on your hosts so you can use TLS vs SSL.

A few days ago, Veeam announced the General Availability of Veeam Backup & Recovery 7.0, which was highly anticipated for the number of features and improvements that it brings. I’m going to do a few posts on the installation and usage of the product to get used to it myself.

1) Download the installation files from Veeam’s web site. You’ll need to register, which will e-mail you a trial key. The installation file comes as an ISO, so you can unpack it or mount it to your VM. When you do, launch the SETUP.EXE

2) Click on the Install for Backup & Recovery

3) Click through the licence agreement, and then select the licence file. The trial comes with 32 sockets.

4) Select the program features to install. PowerShell SDK is not selected by default.

5) Veeam will identify missing pre-requisites, and install them with a single click:

6) Enter the account for the service to run with. I’d recommend a Service Account of some sort.

7) Install the bundled MSDE SQL 2008 R2 instance.

8) Select the default ports:

9) vPower NFS is used to mount backups as an NFS datastore – and your system will need a root folder to mount this in. You also need to select a System Catalog folder for VM’s you will be indexing the contents of – useful if you need to search for files within VM’s in your backup.

10) You’ll get a chance to review the installation summary screen and click INSTALL.

11) That’s it. You didn’t think it was that hard, did you? 🙂 It’s Veeam – it’s that good!

In my previous post, I covered the installation of Veeam B&R v7.0. Now I’ll review the basic setup

1) Launch the program via the shortcut.

2) Click on the BACKUP INFRASTRUCTURE button.

Click on ADD SERVER and then choose VMWARE VSPHERE.

3) Enter the FQDN of your vCenter Server.

4) Enter credentials for the vCenter Server.

I create a service account, “svcVeeamAdmin” that has rights both to vCenter and Domain Admins for internal VM tasks such as VSS, Indexing, etc. Follow whatever best practices apply for your environment.

Name the repository appropriately so you can easily identify it later.

8) Choose the type of Backup Repository. While Microsoft Windows Server and Shared Folder (CIFS/SMB) may seem similar, the Windows Server option allows for the installation of an agent to help with efficiency and a Shared Folder would be more like a Linux server or NAS that is exposing a CIFS share but is not otherwise running Windows.

9) Enter the FQDN of your repository server:

10) Choose the credentials you added previously:

11) When you click NEXT, it will check for the presence of an agent on the destination, and then allow you to install or upgrade.

12) Once the installation completes, you’ll see the summary.

Click Next

13) On the summary screen, click FINISH to complete creating the repository.

14) Click POPULATE to show the drives on the system with the agent installed.

Select the drive you want to use, and click Next.

15) Click BROWSE or type in the PATH TO FOLDER, then click POPULATE.

You can also control the load – the number of concurrent backups or limit the MB/sec (so as to not saturate the network).

If you click ADVANCED:

You get options to align backup blocks and decompress the backups – largely for when writing to a repository that supports its own deduplication. Different use cases will have the remote path obtaining better results than the Veeam deduplication.

16) Next, we configure the vPower NFS folder.

We can select to enable it – or not. Also you can select a Write Cache folder – such as an SSD path, to speed up the writes. Click Next.

17) On the REVIEW page, you’ll see a summary:

Click NEXT

18) And then on the APPLY screen, click FINISH.

19) Click on the Backup & Replication button, and then click Backup Job.

Name your backup job, and click Next.

20) Click ADD to add VM’s to your backup:

21) The default view, when you expand the tree shows your Hosts & Clusters view, with your VM’s – just like the vCenter Client.

I’ll pick my running VM’s, but an alternative might be to use a VM’s & Template’s view, and pick VM folders.

22) Next, we need to pick a Backup Proxy (Used to speed up backups if the Veeam B&R VM cannot attach to them locally) and Backup Repository (to store the backup files). The Backup Repository can be off host, such as a DAS or a NAS that is not part of your Shared Storage – remember not to keep your backups where your primary data is!

Select the REPOSITORY we create earlier.

Note the “Configure Secondary Destination for this job” – this will let you store a second copy on a second repository. We’ll cover that another day, after we make our first backup job.

23) Click the ADVANCED button to get access to the good options:

I personally prefer “Reverse Incremental” which makes the previous night’s backup file the Full. This does require a higher IOPS rate, and there are reasons why other options are better.

I would recommend clicking the check box for PERFORM ACTIVE FULL BACKUPS PERIODICALLY if you want to create a new FULL backup every now and again. It’s up to you if you wish to choose “forever incremental” or have occasional full’s. Its really a matter of confidence.

24) Click on the STORAGE tab:

Here you set the Deduplication, Compression, and optimize the storage method.

25) Click on the NOTIFICATIONS tab:

Put in an e-mail address to send your alerts to. I recommend checking the VM NOTES box and choose a field other than “NOTES”. If you use NOTES the default NOTES field in vCenter Client will get overwritten – and you might be using that for something. Veeam is nice enough to let you pick the field you want to use, and if it does not exist, it will create it.

26) Click on the vSphere tab:

Here, you choose if you want to use VMware Tools quiescing – this will get you “Application Consistent” backups vs “Crash Consistent”. Not using CBT (Change Block Tracking), which only backs up new/changed blocks from the previous backups, would be just silly.

27) Click on the ADVANCED tab:

The defaults are pretty good, but look into the manual to decide if you want to fiddle with this.

28) The STORAGE INTEGRATION tab:

Will allow you to use the SAN snapshot to do the backup. I understand this is limited to just the HP VSA currently, but let’s the SAN snapshots be used, with the assumption that the SAN tools created clean, application consistent quiesced backups.

Click OK.

29) You’ll get a warning that you need to set up e-mail options:

So we’ll go do that in a bit.

30) Back on the NEW BACKUP JOB screen, we go to the GUEST PROCESSING option:

If you’re just looking for the VM’s and crash-consistent backups, then you don’t have to change anything. There are many reasons why you might want this. However, I would check the boxes for Enable application-aware image processing (for VSS to truncate things like Exchange and SQL logs) and then Enable Guest File system indexing (if you want to search backups for particular files). Choose the credentials we created earlier:

If you click the ADVANCED button, you can modify the previous options on a per-VM basis.

Click NEXT.

31) On the SCHEDULE window:

Set up your schedule to run on particular days and times. Veeam has an excellent Automatic Retry option where it will retry failed VM’s at the end of the job. Often this is enough to bypass whatever made the VM fail in the first place – maybe it was being backed up by something else, it was too busy to quiesce, etc. I’ve seen other backup products fail an entire job of 50 VM’s if only a small error occurred and not retry – or retry the WHOLE job.

You also have the option to automatically terminate the job if it exceeds the backup window – such as 8AM in the morning, when staff start showing up.

Click CREATE!

32) On the summary screen, click RUN THE JOB if you wish, and click FINISH.

That’s really about it. I realize at 35 steps, it *seems* like a lot, but that’s not just creating the first job, but setting up the entire system, the repository, the proxy, the notifications, etc. They’re also very easy steps. At this point you have a fully functioning daily backup of your VM infrastructure.

When you click on the job, it will show you details in the lower pane. Depending on available resources (number of CPU’s), Veeam will choose the number of parallel VM’s it can back up. More vCPU’s = More simultaneous VM’s.

Here you can see that I’m getting 81.9MB – a decent rate considering I only have 1GbE link to my media server. More interesting is that it has processed 27.5GB, but only had to transfer 13.8GB – a 2x savings in bandwidth. You can see the deduplication and compression savings live. This rate tends to get increasingly better as additional VM’s are processed, as they likely contain very similar data if they’re all similar OS’s with similar roles or built from similar templates.

And now that the job is done (A different job, so the times are off), you can see that my 8 VM’s took 93 minutes to backup, the bottleneck was actually reading from the source (1GbE NFS, who knew!), processed 325GB, but only transferred 79.2GB – a 3.9x dedupe ratio.

An incremental, run immediately after shows a 38 minute duration that read 245GB of VM’s, only had to read 48.9GB of actual data, and only transferred 1.3GB of it due to deduplication and compression. Imagine what that would be like across the WAN compared to a ‘normal’ backup!

I’ll have more posts on the new Veeam B&R v7.0 coming, but head on over to our local Veeam SE, Rick Byrne for more advanced options – http://rickrbyrne.wordpress.com/

Just saw an e-mail come through with the Veeam Community digest. While I don’t have a link directly to that as a post to reference, there are two important bits of information:

Quoted from Gostev directly:

Our QC has finished smoke testing of B&R 6.1 against vSphere 5.1 RTM code, and based on the first results, version 6.1 will NOT officially support vSphere 5.1. In case you are wondering what B&R functionality specifically brakes after vSphere 5.1 upgrade, our quick testing uncovered at least the following problems. Note that there could be more issues – even in functionality that “seemed” to work – as this was not full regressive testing. Thus, please avoid upgrading to vSphere 5.1 until we release B&R 6.5, which we are aiming to make generally available later this year – really, as soon as we can.

Critical issue warning for those still on VMware ESX 4.1! You must update B&R with the latest build (6.1 patch 1b) before upgrading to the recently released ESX 4.1 Update 3, or all your jobs will start failing (refer to the Known Issues sticky for more details). There are no similar fixes available for previous B&R versions, meaning that if you are not on B&R 6.1 yet, you must upgrade B&R first. This is really something extraordinary – usually, VMware does great job with the maintenance updates – I don’t remember a single one that would break our products. Nevertheless, this once again shows how important it is to test EVERY update before applying into production (or, at least, not jump the update as soon as it is released – but let other people validate it, or run into any issues first).

I have to say, Veeam’s communications and openness continues to impress me. A lot of vendors would keep this information close, or tell folks that it must be something they’re doing that’s wrong, and pass the buck. While some could read the above as Veeam having issues, I see it as them being a part of the community and sharing issues as soon as they’re aware of them. This is part of the reason I really like dealing with them as a vendor!

I keep getting occasional inquiries on how my new and shiny Synology DS412+ is doing, so here it goes. In short, I have been extremely happy with performance, stability, noise level and functionality. But a few days ago, I became even happier, because Synology has released DSM 4.1 firmware update. Apart of many cool new features, they have added support for multiple additional VAAI primitives. Note how 412+ is the only mid-range NAS to get the full VMware VAAI support. I cannot be more happier with my choice! Really hoping they will add Microsoft ODX support soon, too. Heads up though – just like B&R 6.1, looks like DSM 4.1 does NOT support vSphere 5.1 yet.

This has been pretty big on the Twitter this last week, as many bloggers, vExperts, home labbers, etc, have this kind of equipment in their labs, and obviously, are affected by it. Compounded moreso by how popular the equipment became when it was one of the first (only?) in this tier to have VAAI support at such a low entry price.

If anyone knows how to find a link to the Veeam Community Digest online, let me know as I’d really like to share a link directly to that, rather than cut/paste from e-mail. Give credit directly to the source and all that.