2013 was the year we learned we must encrypt our data if we don't want the likes of the U.S. National Security Agency or the U.K. Government Communications Headquarters reading it as it crosses the Internet.

The security industry has the enemy it always dreamed of to help it make the case for encryption adoption, but users looking to secure their data and communications need to be wary of claims made in marketing messages. Securing data in motion is the priority, experts say, and some large Internet firms are already making progress in this area, but encrypting data at rest without losing its usefulness will prove a greater challenge.

"The NSA's surveillance has opened the eyes of many people around the world," Lamar Bailey, director of security research and development at security firm Tripwire said via email. "Security professionals have always known that this style of surveillance is possible with the right resources, but this episode has been a big wake-up call for everyone. Many countries and companies outside the U.S. are now taking a harder, more in-depth look at software and hardware that comes from the U.S., although the silver lining is that mainstream users are now more concerned with encrypting data and reviewing how their information is being shared."

The public debate sparked by the surveillance revelations in recent months has prompted some encouraging responses already: Google has encrypted the links between its data centers; Yahoo is working to do the same and has promised to enable SSL encryption by default for webmail and other services, and Twitter has enabled an SSL feature called forward secrecy, already implemented by Google and Facebook, which makes mass decryption of SSL traffic hard even if the website operator's master private key is compromised.

These and other examples send a clear message: securing the data transport channels to prevent unwanted upstream interception is a priority. The Internet Engineering Task Force, an organization that develops Internet standards, is already working toward this goal. Together with other Internet infrastructure groups, IETF expressed concern that the reported mass monitoring and surveillance by government agencies undermines the trust and confidence of Internet users globally.

The technical community should work to make eavesdropping expensive and force the NSA to abandon wholesale collection of data in favor of targeted collection, renowned cryptographer Bruce Schneier said during the technical plenary at the 88th IETF Meeting in November.

Matthew Green, a cryptography professor at Johns Hopkins University in Baltimore, thinks all communications should be encrypted by default.

"There's really no excuse for sending cleartext data over the Internet anymore," he said via email. "Almost every product is powerful enough to support encryption, and the software to do it is freely available and built into many operating systems. Aside from a few edge cases, the only reason not to encrypt is pure incompetence."

Securing data in transit is a good start, but it is mostly large companies with a lot of resources and know-how that have taken the initiative so far. The majority of efforts seem to come from cloud service providers, not packaged software developers, and the question of whether smaller vendors are able to properly implement encryption remains open.

"Finding software developers who understand IT security is rare enough, and employers don't usually hire security-skilled programmers because they're more expensive," said Raoul Chiesa, president of security consultancy firm Security Brokers and a member of the permanent stakeholders group at the European Network and Information Security Agency. "This means that speaking about built-in encryption to low-level coders is just not feasible," he said in an email.

Fortunately, the NSA's vast resources and capabilities can act as an incentive to implement encryption in a secure way.

"One way to look at the NSA is that they're the adversary we've always dreamed of," Green said. "If there's a practical exploit in a piece of encryption software, you can be pretty sure they're thinking about how to exploit it; and they have the technical capability to do so," he said.

There's a huge paradigm shift from the way things were before the surveillance leaks by former NSA contractor Edward Snowden. Then, the attitude was: "Hey, weaknesses are OK, since nobody's smart enough to exploit them," Green said.

As companies rush to implement encryption to keep up with competitors and respond to market demand, some may fail to do it properly, warned Mark Bower, vice-president of data protection vendor Voltage Security: "End users need to be careful about claims of security, especially anything proprietary in nature, or claims of 'military grade' without any actual validation." Such claims may later be found flawed and useless, he said.

"Often the implementations are not well thought through, especially the key management or the implementation of encryption and hashing," he said. One example he alluded to was that of WhatsApp Messenger. In October a security researcher reported that the popular mobile messaging application contained a basic mistake in its encryption implementation, making it possible to easily decrypt intercepted messages.

Tim Erlin, director of IT security and risk at Tripwire, said it's easy to make mistakes that can render encryption ineffective: "It doesn't matter if your database is encrypted if the application used to access it is available with a default password."

To catch such errors, there is a need for more transparency. Many encryption standards are open, but without third-party audits or public review of code, it's difficult to know whether products using them contain accidental or intentional backdoors.

Transparency is critical for trust, and vendor claims aren't enough, according to Bower. "A vendor can't simply tick a box and say encryption is turned on," he said. "That means nothing. It's how it's used, when data is encrypted and decrypted, with what method, and the process by which keys are managed, stored, and restored which is critical to understand before there's any measure of benefit for users. If it is well implemented, and provably so, then data privacy and surveillance risks are dramatically reduced."

RSA, the security division of EMC, recently found its trustworthiness and transparency called into question following a report by Reuters that it was paid US$10 million by the NSA to make a flawed pseudorandom number generator called Dual_EC_DRBG the default choice in its BSAFE cryptography library. Random number generators serve a critical purpose in cryptography and using a weak one can undermine the security of the whole crypto system. RSA denied that it had ever entered into a contract or project with the intention of weakening or introducing backdoors into its products.

The company had advised customers to stop using Dual_EC_DRBG in BSAFE following media reports in September claiming the NSA pushed this flawed random number generator as a standard as part of its efforts to defeat encryption.

Bower believes the recent surveillance revelations should act as a call for independent verification of security claims and cryptographic system designs in both open-source and commercial products.

The current situation with many commercial products "motivates users to consider existing open-source alternatives that might provide greater transparency and security," according to Erlin, while Bower expects the open-source software community to start pushing for open audits to make sure there are no backdoors in popular free tools.

An example of that is the project organized by Green and Kenneth White, principal scientist at health software-as-a-service provider BAO Systems, to audit TrueCrypt, a popular open-source disk encryption tool.

Snowden's revelations of mass surveillance have brought the value of encryption into focus for a large number of people, but the challenge of securing all communications involves making encryption seamless and fully automated.

It needs to be as easy as using electricity, Bower said. "End users don't need to understand electromagnetic theory to use electricity; they simply plug in, turn on, and use their favorite new gadget."

Bower believes advances in encryption technology during the past decade have made that possible, but others remain skeptical.

"We will continue to see both open source and commercial solutions pop up that claim to be fully secure and immune to any surveillance attempts," said Jake Kouns, chief information security officer at security consultancy firm Risk Based Security. "My concern is that in most cases these services will be for niche users only."

Expecting people to encrypt all communications is unfeasible, but it would be a start if they would at least encrypt sensitive communications, said Carsten Eiram, the chief research officer at Risk Based Security. "Many even fail to do that, which is likely due to people simply considering it too much of a hassle to import and manage keys," he said.

However, this shouldn't deter developers from adding encryption functions. Users will understand their importance and will learn how to use them, Chiesa said.

Encrypting "data in motion" is actually the easy problem, according to Green. "The next frontier is encrypting data at rest. Right now this is difficult, since companies need access to your data if they're going to compute on it -- think Google searching your email. We don't have good cryptographic solutions to this problem, and frankly a lot of this data is available in cloud environments where the security is unknown at best."

Most businesses understand they should secure their data using encryption, but when it comes to implementing it things usually come to a halt, Kouns said. "For most small to mid-size businesses, implementing encryption is a daunting task that is too complicated and expensive."

Even with significant problems to overcome, most security experts believe that the trend of building encryption into products and services will continue to increase in both the consumer and business markets.

"Anyone bringing out a new service using consumer and analytic data, whether in the cloud or not, is going to struggle without a data protection strategy," Bower said.

"I expect to see companies continue to invest in this space," Kouns said. "The vendors and services that solve this issue in a simple and cost effective manner are the ones that will benefit the most."

"Encryption can make the difference in a service," Chiesa said. "See for example the old Megaupload and the new Mega: user privacy is the center of the whole thing now."

Copyright 2016 IDG Communications. ABN 14 001 592 650. All rights reserved. Reproduction in whole or in part in any form or medium without express written permission of IDG Communications is prohibited.