Ready for anything

The 2006 war speeded up Beirut-based Soft Solutions' efforts to put a disaster recovery system into place.

The 2006 war brought Beirut-based Soft Solutions' backup needs into sharp focus, and speeded up its efforts to put a suitable disaster recovery system into place.

Service providers the world over are always fearful of the potential for data loss resulting from a disaster, but for those operating in countries such as Lebanon, where the threat of armed conflict is currently high, disaster recovery ceases to be an abstract possibility.

The clients started to call us, and ask: 'When will you implement the code?' 'Is the code in a safe place?'

Soft Solutions is one firm which has had to deal with this reality. A French company, Soft Solutions moved its entire datacentre and programming operations to Beirut in 1999 - it also has offices in New York and branches in the UK and Germany, as well as its French headquarters and Beirut site.

"Being in Lebanon is very challenging, as people know, especially after the July 2006 war. The clients started to call us, and ask: 'When will you implement the code?' 'Is the code in a safe place?' 'Will this affect us?' 'Are you functioning?' 'What is your disaster recovery plan?' - all these kinds of questions," says Mounir Rahmeh, managing director of Soft Solutions.

"We had to calm the clients - don't forget, we have penalties that we get if we don't implement on time or deliver services according to the SLA. It was really challenging," he adds.

The company produces custom applications for retail operations, and has a global customer base including Carrefour, CVS Pharmacy in the US, B&Q in the UK, and Majid Al Futtaim (MAF) Retail in the UAE.

Until 2006, Soft Solutions had a manual backup regime, according to Rahmeh: "We had two IT teams which would spend all day doing backups on servers Previously, we held the backups in different places - sometimes at a bank, where we had an account, sometimes at my place! The headquarters in Lebanon has two buildings - in one of the buildings are all the IT operations, and the other building is facilities for visiting teams from Europe and the US - apartments they can use, and so on. We also used to backup to a server in one of these apartments - again, this was manual."

The company started considering how to improve its backup and disaster recovery (DR) plan in 2005, but it did not start implementing a new system until August 2006 - after the war with Israel.

But although the war was a major element in pushing Soft Solutions to accelerate the adoption of a new DR system, the deciding factor was a specific demand from a client.

"The real need came when our clients decided to audit our operations - two of our main clients wanted to visit the lab and do an audit of our continuity plan. This was when we implemented the new system," says Rahmeh.

"We created three backup sites: one is a one-hour drive from our office in Beirut - it's in northern Lebanon, on hosted servers. Then we decided we needed something outside Lebanon, but not too far away - we thought about Cyprus, and we decided to host two servers with Cytanet, one of the main ISPs on the island. It's actually one of the departments belonging to the telecommunications department, at Nicosia. The third backup site is in France, at the global headquarters," he explains.

The system Soft Solutions chose for the backup operations was CA's XOsoft, for which Rahmeh is full of praise, and which he credits for the speed of implementation: "It's quite funny - we thought this would take us months to accomplish; we were able to establish the whole thing within one week, with the help of CA. The support that we got from CA - preparing for the installation, checking the information when we did a restore, to check it had been correctly backed up. It was helpful to have a team from CA's partners in Lebanon - Triple C - with us."

After implementation, the key aim for Soft Solutions was to bring the client in and give a demonstration of its disaster recovery abilities. Rahmeh acknowledges that this took "a lot of rehearsals", but the end result was a complete success.

Rahmeh explains: "The first day the client arrived, I took three of my engineers to the local site north of Beirut - we were able to connect to the backup servers, and we were fully functional and connected to the client's code within three hours.

"The second day, we went to Cyprus - we took the plane at 7am. We went to Nicosia, to Cytanet's headquarters - we can use their facilities as part of the contract. Using their laptops the team was able to connect to the servers within five hours. I would say that we were really functional in Cyprus within ten to 12 hours, counting the plane, arrival at Nicosia, and installation," he adds.

It's critical for Soft Solutions to restore systems operations as quickly as it possibly can, thanks to the stringent SLAs it has with certain clients, says Rahmeh: "According to the service level agreements we sign, sometimes we have to deal with a defect in the applications within 24 hours. Whenever there's a client reporting a problem, we need to be able to find the solution and send the patch out quickly."

Soft Solutions's major technical challenge was bandwidth. While the links within Lebanon were fast, international links were more of a concern, along with the risk that they could fail.

The firm needed to make sure its backup system was as efficient as possible, and that its external links could handle the backup data a week reliably.

The firm was able to tune XOsoft to use a particular proportion of the available bandwidth for the bandwidth, and also to back up only the changes to the code base, instead of the whole thing.

Rahmeh says another key advantage to XOsoft is its' ability to back up changes regularly, instead of holding on to a large amount of data and backing it up in one go.

Regarding the external links which need to be able to handle 1.5Gbytes of data a week, Rahmeh says: "Being in Lebanon, international connectivity is always a potential problem, so we have to be prepared. We have a leased line connection, and a satellite connection, which we can use as a backup to upload and download. Our main connection is the leased line, then we have the satellite for backup - and after that, if the satellite goes down, it's a major disaster for everyone!"