Category Archives: Technology Blog

The recently announced SSL heartbleed bug has the potential for being one of the most significant security threats that the Internet has faced in the last decade. This bug affects about 60% of the servers on the Internet and, although unlikely, could have resulted in the loss of every password and private key used on these systems!
What is known Today

A hacker exploiting this bug can potentially obtain the user name and password of anyone who has logged on to a vulnerable server. They can also obtain the private keys used to verify the authenticity of the system itself.

Knowledge of this exploit was not widely known until this week but there is evidence that it was known and exploited by some in the hacker community before the announcement was made. How wide the compromise has been is currently unknown. Complicating matters is the fact that this exploit leaves no evidence of intrusion in any logs making it unlikely that the extent of past compromises will ever be known.

Patches have been released by many vendors (and many major sites have already been patched but there are many products for which confirmation of the vulnerability is not yet known.

While a few major sites have already installed patches, the majority of affected systems remain vulnerable at this time.

Any account that has been used on an unpatched and vulnerable system during the last two years has been potentially compromised.

What should users of affected systems do?

DO NOT LOG IN TO VULNERABLE SITES UNTIL THEY HAVE BEEN PATCHED!

DO NOT CHANGE PASSWORDS UNTIL YOU CAN VERIFY THAT THE SITE HAS ALREADY BEEN PATCHED FOR THIS BUG!

This bug is now widely publicized and widely known in the hacker community. Exploits of this bug will escalate exponentially until systems are patched and any new login or password changes carry a far greater risk of compromise!

When a vulnerable system has been patched, change your password. If you have used the same password on any other sites (even if the other site was not vulnerable) change those passwords too.

What should be done by those in IT who manage vulnerable systems?

Identify all vulnerable systems.

Any system that uses an unpatched openSSL libraries (version 1.01) are vulnerable. This includes webservers, VPN servers, TLS/SMTP encryption, management interfaces on appliances, servers, etc…

Immediately patch any vulnerable system (beginning those that are exposed to the Internet).

Remember cloud based applications used by your users are also potentially vulnerable and any IP stored on these systems is also vulnerable.

After patches have been applied, expire ALL passwords on affected systems.

Audit all accounts on affected systems; verifying that no new accounts have been created.

WARNING: The longer a system remains unpatched, the more potential there will be for a compromise. Once a system has been compromised a hacker can add backdoor accounts, root kits, etc… that will help them maintain access after patches have been installed!

Note: The tests for the Heartbleed vulnerability can only tell if a server is currently vulnerable. If you logged into a machine before it was patched you should still change your password. Additionally, when pools of servers are providing services, it is important to wait until all of the servers have been patched before attempting a login or password change.

We have all seen facebook posts similar to the one with a cute little girl holding a sign that reads “My dad said that if I get a million ‘likes’ he will buy me a puppy.” While these posts appear innocent, they rarely are. Most of these posts are intended to gain enough “likes” to raise Facebook’s score for the page associated with the post. The higher score these pages have gained allows these pages to have much broader distribution when they post the subsequent advertising; sometimes these pages are offered for sale to other advertisers looking for the broad distribution of their adds (even though this is prohibited by facebook’s policies). Realize that most of the time you are not helping a little girl get a puppy, encouraging a child with cancer that people care for him, or encouraging a handicap person to run a marathon; you are allowing yourself to be targeted with advertising you almost certainly did not want. While participation in a “like” harvesting scheme will result in nothing more than receiving annoying (and possibly offensive) ads, other internet SCAMS can have much more severe consequences. Always realize that there just may be malicious motives behind the seemingly innocent requests you receive. In the anonymous world of the Internet, it is always best to approach requests from those you do not personally know with a great deal of suspicion.

What application or applications will be supported on this storage platform?

Understanding requirements of the application or applications placed in service on the storage platform is a critical piece of information needed when choosing a storage platform. Probably the best way to illustrate this issue is to look at a storage platform like EMC’s Isilon platform. It scales extremely well when presenting NAS volumes to hundreds or even thousands of clients. Performance increases almost linearly with each added shelf because the I/O is distributed across all of the nodes (shelves) in the cluster. The size of existing volumes can be expanded or reduced without taking volumes offline and shelves can be added or removed from production clusters without service interruption; this allows for tremendous flexibility in adapting to rapidly changing storage needs. However, if you intend to run a large OTP database, Isilon is probably the wrong storage platform for the job. The Isilon platform provides reasonably good throughput to each client and can sustain that throughput to a large number of clients simultaneously, but it is not a great choice when very high performance is needed for only a couple of clients (something typical of OTP databases). Additionally, most OTP databases are going to perform best when a LUN is presented via a SAN infrastructure but Isilon performs best when presenting NAS filesystems.

Do multiple applications intended for use on this storage platform have significantly different requirements? Some storage platforms perform well in a SAN configuration but poorly in a NAS configuration, others perform well in a NAS configuration but poorly in a SAN configuration.Some perform well in one configuration or the other but not both simultaneously, and some storage platforms do a reasonably good job when simultaneously publishing both SAN and NAS volumes. Knowing how you intend to use your storage can help narrow the choices considerably.

What are the performance requirements needed for your applications?

One important, but often overlooked, aspect when evaluating performance needs of a storage platform is the labor costs incurred by a slow system. When end users incur delays that inhibit their ability to complete their work quickly, labor costs increase. Understanding how storage performance impacts the interactive performance of the application utilizing that storage is critical to evaluating your performance needs. For example, if a $50,000 investment is needed to double the interactive performance of an POS application and the current response time is .1 seconds and the increased performance results in a new response time of .05 seconds, the investment in the higher performing system was probably a waste of money. However, if the current response time is 30 seconds and a faster storage system will reduce it to 15 seconds then the investment may be warranted. If this system supports a call center staffed with 100 employees taking orders and if the average order is $20 and the time savings due to the increased interactive performance allows for 4 additional calls per day / per employee. The daily ROI would be $8000 dollars per day with the initial investment paid for in only 7 days! Even if the cost for a faster storage platform in this example had been $500,000 it would have still been a very smart investment.

What are the recurring costs for administering the storage?

Do I already have staff trained to administer this storage platform and what will it cost to keep my staff current as software is updated? Continued training across multiple storage platforms can be costly and a lack of training can be even more costly if it leads to a critical mistake.

What are the recurring costs for H/W and S/W maintenance and (very important) will these costs change significantly in the 2nd year, 3rd year, 4th year, etc…?

Finally, it is important to remember that many considerations we make when choosing a storage platform are interdependent. For example, a comprehensive report detailing the expected ROI for different storage platforms may open the “purse strings” when it comes to budgeting. A need for extremely High Availability and Disaster recovery may lead to compromises on performance, the applications we use may have better support on certain vendor’s storage platforms (especially related to backups and cloning), etc… It is critically important to consider all aspects of your computing environment before choosing a storage platform.

A brief overview:

Choosing a storage platform for your computing environment is one of the most critical decisions to be made and one of the most difficult. There are many different approaches to solving storage problems and each has its benefits and its drawbacks. Understanding the specific needs for your project is a critical first step in the process of choosing a storage platform. I would like to explore some of the key considerations that should be understood before “shopping” for new storage begins.

What is the budget?

Storage platforms can span a very wide range of costs. Simple platforms can be obtained for only a couple of thousand dollars but enterprise class systems can cost millions. Knowing your budget before calling a vendor can help you and your sales person avoid wasting time looking at systems and features you cannot afford.

Remember that storage platforms are usually heavily discounted so it is important that you do not base your choices on the list price of the system. At times I have seen discounts well over 60% when the vendor is hungry for the sale. For this reason alone, it is usually unwise to look at storage from only one vendor; when there is competition, there is a much better chance at getting the best deal.

What are the Service Level requirements that must be met?

Fault tolerant storage platforms can be very expensive and so it is very important to understand what SLA’s are required before searching for a storage system. In an environment where most business can be conducted “offline” in the event of a system failure, a basic storage platform with next day parts replacement may be suitable. However, in an environment, like a large OTP database, where tens of thousands or sometimes millions of dollars can be lost each hour the system is down, the cost of highly fault tolerant storage is just part of the cost of doing business; these costs are a drop in the bucket compared to the losses that could potentially be sustained in the event of a storage system failure.

When remote replication is required there are a number of considerations that come into play when deciding on the appropriate technology.

What are the bandwidth requirements needed to replicate the data and what are the recurring costs associated (Budget considerations again).

How much data loss is acceptable in the event of a failure that requires a move to the DR systems. Many replication strategies update the remote site only periodically i.e. once an hour, once a day, etc… If a storage system were to fail or be destroyed in a fire, flood, etc… near the end of the update window, all data updated during that window would be lost.

Is replication of your applications best handled by storage replication or replication at the applications level? While storage vendors will always want to sell their replication, often the best strategies for updating database applications are done at the applications level and not at the storage level because the recurring costs of the WAN tend to be less and the time window for potential data loss is much shorter.

When considering the requirements for data protection, it is important that one not forget about backup and restore strategies. Storage platforms today are often an integral part of any backup and recovery process. Here are some considerations regarding backup and recovery.

Is there a time window for backups that is sufficient to complete required backup. For applications that must be backed up while online, storage snapshot support is often required. Additionally, supplemental software is often also required to ensure that an application’s data on disk is in a “backup ready” state when the snapshot is initiated. These features often require supplemental licenses at additional costs.

Are online periodic snapshot backups required? Many sites take periodic snapshots during the day to ensure that a minimal amount of data loss is incurred if a file or database becomes corrupted. Note: online snapshots are never an appropriate substitution for regular backups because a single raid failure can cause all snapshot data to be lost simultaneously. Most snapshot data is composed of a single copy that is shared between all snapshots.