Security researchers and crypto experts have spent the last few days trying to figure out the details of a recently announced backdoor in Juniper NetScreen firewalls that could allow attackers to decrypt VPN (Virtual Private Network) traffic. They believe that they found the answer: a combination of likely malicious third-party modifications and Juniper's own crypto failures.

According to experts, Juniper was using a known flawed random number generator called Dual_EC_DRBG as the foundation for cryptographic operations in NetScreen's ScreenOS, but believed it was doing so securely because of additional precautions it had taken. It turns out those safeguards were ineffective.

The VPN decryption issue was announced by Juniper Thursday along with another vulnerability that could provide attackers with administrative access to NetScreen devices through the use of a hard-coded master password. Both issues were the result of unauthorized code that was added to ScreenOS and were discovered during a recent internal code audit, the company said at the time.

Crypto specialists delved into the VPN issue, whose description made it more appealing to them, as the ability to spy on encrypted traffic is always a big deal.

It didn't take long for someone to notice that Juniper's latest patches reverted a parameter back to a value that the OS used before version 6.3.0r12, the first in the 6.3.0 branch that Juniper claims was affected by the VPN decryption issue.

According to further analysis by Ralf-Philipp Weinmann, founder and CEO of German security consultancy firm Comsecuris, that parameter turned out to be Q, one of two constants -- P and Q -- that are used by the Dual_EC random number generator (RNG).

Dual_EC was standardized by the U.S. National Institute of Standards and Technology (NIST) in 2007 after being championed by the U.S. National Security Agency, which played an important role in its development. Shortly after, Dan Shumow and Neils Ferguson, two researchers from Microsoft, disclosed a major weakness in the standard that could serve as a backdoor.

"Omitting the mathematics, the short version is that Dual EC relies on a special 32-byte constant called Q, which -- if generated by a malicious attacker -- can allow said attacker to predict future outputs of the RNG after seeing a mere 30 bytes of raw output from your generator," said Matthew Green, a cryptographer and assistant professor at Johns Hopkins University, in a blog post Tuesday.

The default value for Q in the NIST specification was generated by the NSA and there's no guarantee that it was done in a random manner. In fact, in September 2013, based on secret documents leaked by former NSA contractor Edward Snowden, the New York Times reported that the NSA intentionally engineered the weakness.

The fallout from this report prompted NIST to retire Dual_EC_DRBG from its recommendations and to advise users to transition to other random number generators. After the NIST advisory, Juniper admitted that ScreenOS used the Dual_EC_DRBG, but claimed that it did so "in a way that should not be vulnerable to the possible issue that has been brought to light."

Instead of using the P and Q constants recommended by NIST, which are supposed to be points on an elliptic curve, ScreenOS uses "self-generated basis points." Furthermore, the output of Dual_EC is then used as input for another random number generator called FIPS/ANSI X.9.31 that's then used in ScreenOS cryptographic operations, the company said at the time.

It's unclear why Juniper decided to use one RNG's output as input for another RNG instead of using the second, and more secure, RNG directly. However, if that implementation was accurate, controlling Q wouldn't be of much use for an attacker. So why did the supposed attackers go to the effort of changing the parameter?

The answer comes from an independent researcher named Willem Pinckaers who, while reading Weinmann's analysis, noticed an error in Juniper's code that was supposed to pass the Dual_EC output to the ANSI generator, causing that step to fail.

"Willem Pinckaers pointed out that the reseed_system_prng function sets the global variable system_prng_bufpos to 32," Weinmann said in an update to his blog post. "This means that after the first invocation of this function, the for loop right after the reseed call in system_prng_gen_block never executes. Hence, the ANSI X9.31 PRNG code is completely non-functional."

The error appears to predate the unauthorized changing of the Q point by unknown attackers and can be viewed as a backdoor itself. Weinmann actually referred to the whole issue as the "backdoored backdoor."

"To sum up, some hacker or group of hackers noticed an existing backdoor in the Juniper software, which may have been intentional or unintentional -- you be the judge!," Green said. "They then piggybacked on top of it to build a backdoor of their own, something they were able to do because all of the hard work had already been done for them. The end result was a period in which someone -- maybe a foreign government -- was able to decrypt Juniper traffic in the U.S. and around the world. And all because Juniper had already paved the road."

Juniper did not immediately respond to a request for comment.

According to Green, even if Juniper did not intentionally introduced this vulnerability, it is a great example of what could happen if law enforcement authorities in the U.S. get their way and force vendors to backdoor their encryption implementations so that communications can later be lawfully intercepted.

"The problem with cryptographic backdoors is not that they're the only way that an attacker can break intro our cryptographic systems," Green said. "It's merely that they're one of the best. They take care of the hard work, the laying of plumbing and electrical wiring, so attackers can simply walk in and change the drapes."

"What this vulnerability tells us is that these concerns are no longer theoretical," he said. "If encryption backdoors are indeed in our future, then all of our jobs just got harder."

To comment on this article and other PCWorld content, visit our Facebook page or our Twitter feed.