In this blog post I’ll share how to setup Cloudian HyperStore 7.2.1, create an immutable bucket and how to make use of it in Veeam Backup & Replication v10 for testing purposes and getting your hands dirty with object lock which in essence will provide ransomware protection but first let me talk a little bit about why this is a big deal.

One of the most interesting and useful features in Veeam Backup & Replication version 10 is the “replacement for tapes”-feature called Immutability. Why does it replace tape I hear you ask. A few different reasons but the biggest to me is that you can protect your data from being tampered with just as with tape but unlike tape your data is still online and accesible. It’s built on an AWS S3 feature called Object lock, eventhough it’s originally from AWS that doesn’t mean that it’s only available for AWS users. There’s actually a growing list of object storage solutions and vendors implementing the latest AWS S3 API, both as cloud based solutions but also for on-premises solutions using either a hardware- or a software approach. Today the object lock functionality is supported by AWS S3 of course but also Zadara VPSA Object Storage (v 20.01 or later), Ceph (v14.2.6 or later) and Cloudian (v 7.2 or later). You can find the ever growing list of compatible object storage solutions that works with Veeam on this unofficial list which includes both object storage solutions with and without object lock functionality.

So let’s get back to basics, what exactly does the object lock feature do? Well, the short answer is that it write protects the data you save to on object storage solution for a period of time which you can define, making the data accessible to read (meaning online) but it cannot be changed or deleted untill that time has passed (so basically the equivalent of an offline tape but instantly accessible to recover from). “WORM” escentially, Write Once Read Many. Your data is still online and ready to be used if you need it but if your hit by some ransomware or malicious admin/hacker the data cannot be changed or deleted.

Want to kick the tires? Give it a spin? What you need: Veeam Backup & Replication v10, An Object Based storage solution, in this post I’ll be using Cloudian HyperStore, and an lab environment to deploy it all on. You will also need the AWS CLI when creating the S3 bucket in the Cloudian environment.

I’m going to use a few new hostnames in my lab evenrionment, so first thing is to add those to my DNS server (I’m using a zone called vcsp.local where I’ll add the records):

cloudian01 / 192.168.50.231

cloudian02 / 192.168.50.232

cloudian03 / 192.168.50.233

cmc / 192.168.50.231, 192.168.50.232, 192.168.50.233

iam / 192.168.50.231, 192.168.50.232, 192.168.50.233

s3-nordics / 192.168.50.231, 192.168.50.232, 192.168.50.233

s3-admin / 192.168.50.231, 192.168.50.232, 192.168.50.233

s3-website-nordics / 192.168.50.231, 192.168.50.232, 192.168.50.233

sqs / 192.168.50.231, 192.168.50.232, 192.168.50.233

By adding a hostname with multiple IP address you will get a basic “load-balancer” distributing connections between the different nodes. It’s not a real load-balancer but the DNS server will resolve a hostname to a new IP address for every request it gets, this is called DNS Round-Robin. If you don’t have a DNS server you can install a lighweight DNS server as part of the Cloudian deployment called dnsmasq, when you get to the part where you install HyperStore (the step below that says “. /cloudianInstall.sh force“), replace it with the command “. /cloudInstall.sh dnsmasq force” instead.

Configure the three nodes:

Use Deploy OVF template in vSphere to install the Cloudian HyperStore OVA. Import the Cloudian HyperStore OVA 3 times using the names Cloudian01, Cloudian02 and Cloud03.

Change resource of the VMs to 8 vCPU and 16 GB RAM (which is the minimum requirement but in this limited test environment it works with fewer resources, I’ve tested with 2 vCPU and 8 GB RAM and it seems to work ok)

Power on VMs

Logon as root / password using the consolecd CloudianTools

Run the following command and follow the steps./system_setup.sh

1) Configure Networking

1) Ens160

Change IP address to static IP address

Set IP address, subnet mask, default gateway and DNS

Do you wish to save these settings? Yes

Will <IP address> be the address you use for hyperstore-ova in you survey file? Yes

Would you like to restart this interface to activate this new configuration? Yes

P) Return to the Previous Menu

D) Change Domain name

Do you want to change your domain name? Yes

New Domain Name: vcsp.local

H) Change Hostname

Do you want to change your hostname? Yes

New Hostname: cloudian01(cloudian02 and cloudian03 for subsequent nodes)

N) Restart Networking

Are you sure? Yes

P) Return to the Previous Menu

2) Change Timezone

8) Europe

45) Sweden

Would you like to save this timezone setting? Yes

5) Change root Password

1) Change root Password

New Password: <new root password>

Retype New Password: <new root password>

P) Return to the Previous Menu

D) Download HyperStore Files (this step is only needed on the first node you install, i.e. cloudian01)

Select an option to download(if you only want to test object storage choose 1 but if you want to test object lock/immutability feature you need to choose 2)

1) Download HyperStore GA files (HyperStore 7.1.7

2) Download HyperStore EA files (HyperStore 7.2.1, required for Object Lock testing which is a licensed feature and not part of the trial license)

P) Return to the Previous Menu

X) Exit

Continue to cloudian02 and do the same steps above and then cloudian03

Install Cloudian HyperStore On Master (cloudian01 only):

On you PC/Mac using for instance WinSCP or CyberDuck, transfer your license file cloudian_<numbers>.lic to cloudian01 (there’s a trial license included in the download from the previous step that could be used but it doesn’t include the object lock functionality, you need to contact Cloudian to get a license to unlock that feature.

If needed, transfer the license file to /root/CloudianPackages/(or use the one already present in that folder, but again: it doesn’t provide Object Lock functionality)

Logon to VM using root / <new root password>

cd /root/CloudianPackages

./CloudianHyperStore-7.2.1.bin cloudian_<numbers>.lic

cd /opt/cloudian-staging/7.2.1

./system_setup.sh

4) Setup Survey.csv File

Would you like to create a survey file now? Yes

Would you like to add entries now? Yes

Region Name: nordics

Hostname: cloudian01

IP Address: 192.168.50.231

Data Center Name: DC1

Rack name (all nodes in a DC must use same rack name): rac1

Internal Interface (optional): <skip this option by pressing enter>

Would you like to add another entry? Yes/No (you can have a 1 node test bed if you like or add 2 additional nodes if you’d like to test object lock)

P) Return to Previous Menu

S) Script Settings

10) Generate SSH Key File

Install public key on cluster nodes? Yes

Enter password for each node

P) Return to the Previous Menu

6) Install & Configure Prerequisites

1) Install & Configure Prerequisites

Would you like to perform this on all nodes listed in you survey file? Yes

P) Return to the Previous Menu

R) Run Pre-installation Checks

1) Quite mode: (show only warning or failed tests)

R) Run Pre-Install Checks

You will most likely get warnings that you running a virtualized environment, perhaps not enough nodes in the cluster and not enough resources allocated to the VM. If other issues show up as failed they need to be addressed before proceeding.

P) Return to the Previous Menu

W) Write sysctl Configuration

X) Exit

A reboot is required to apply some of the changes. Reboot cloudian01 now? Yes

Wait for cloudian01 to reboot

Logon as root / <new root password> using console or ssh

cd /opt/cloudian-staging/7.2.1/

. /cloudianInstall.sh forceThe “force” switch is required since we’re not using the recommended minimum resuorces, use “./cloudianInstall.sh dnsmasq force” if you need a DNS.

Click the top link displayed: No Storage Policies have been defined. Please create a Storage Policy to create a storage policy

Click + Create Storage Policy in the top right corner

Give it a Policy Name and accept all other default settings

Click Save

Click Users & Groups in the top bar

Click Manage Groups in in the top

Click + New Group

Give it a Group Name: Backupusers

Click Save

Click Manage Users in the top bar

Click + New user in the top right corner

Give it a User ID: veeam_backup_user

Add a password (min 9 chars)

Assign the Group Name created in earlier step: Backupusers

In the field “Search For A User By ID:” type veeam_backup_user and click search

Click Security Credentials for the user veeam_backup_user

Copy Access Key ID and then click View Secret Key to access and copy it for use later

Click Close

Sign out of the CMC GUI

Enable object lock on Cloudian

For more detail on Object lock you should read the document Cloudian-QuickStartGuide-Object-Lock.pdf, below is a summary of the steps outlined in that document that needs to be taken

Log into the Puppet Master node (should be cloudian01) as the root user.

Check to confirm that the HSH is currently disabled.

[root@cloudian01]# hsctl config get hsh.enabledFalse

Set hsh.enabled to true.

[root@cloudian01]# hsctl config set hsh.enabled=true

Push the configuration change out to the cluster.

[root@cloudian01]# hsctl config apply hsh

Confirm that HSH is now enabled.

[root@cloudian01]# hsctl config get hsh.enabledTrue

HSH is now enabled in your system, but no users are yet able to log into it. To provision the default admin user for HSH do the following steps:

log into the CMC as the admin user with password public

Change the “admin” user’s password. in the top right corner, Admin->Security Credentials. This password change causes the system to create a corresponding HSH user.

Once an HSH user has been created, that user can use SSH to log into any HyperStore node. Prefix sa_ should be applied to the admin account when logging on, so user should be sa_admin and password should be <new root password>. The prompt will appear as follows:sa_admin@cloudian01$

You can confirm that you are in the HyperStore shell by typing help:sa_admin@cloudian01$ help

Type exit and press enter to end session

Log on to cloudian01 via console or ssh using root with <new root password>

To disable root password access to all HyperStore nodes:

cd /opt/cloudian-staging/7.2.1

./cloudianInstall.sh

4) Advanced Configuration Options

m) Disable the root password

Do you wish to disable the root password on all Cluster nodes? Yes

X) Return to Main Menu

X) Exit

Type exit and press enter

Try to logon again using console or ssh and verify the root is no longer able to logon.

To create a bucket with object lock this must be done using an API or using AWS command line interface, it can’t be done from the Cloudian CMC.On a management PC, download AWS S3 CLI to create a bucket with object lock.

Make sure to click “Copy backups to object storage as soon as the are created!

Create a backup job and use Scale-Out backup repository above as target, and start it.

Verify Object lock

Once backup job is finished

Go to Home

Under Backups find Object storage

Right click the backup job and select delete from disk

If everything is configured correctly you should get a failed attempt!

By using Veeam Backup & Replication version 10 in combination with a Scale-Out backup repository including an object based storage solution, we can make sure that our valuable data is protected, we get 2 backup copies automatically when using copy-mode, we get a second media type and we get a write protected copy with the immutability option. So in a single job we can actually adhere to the design princple we’ve talked about for a long time called the 3-2-1-rule. How cool is that!

So what we’ve now established is a solid solution that will protect your data no matter if it’s from malicious insiders or ransomware!

In the next blog post will be a follw up on this post where I’ll show you how easy it it’s to recover from a disaster including the Veeam Backup & Replication server, a total site failure. I’ll show you that as long as you have a copy of you backup available in object storag solution (not part of the site that filed of course), you can recover!