It seems that GCM mode encryption has a clear advantage over CBC + HMAC in the sense that it only requires a single key. But it seems that there are some experts here that do not trust it enough to recommend it. This question is a call to those experts to clearly define those concerns.

Are there any properties of GCM mode encryption that make it less secure in practical operations than CBC + HMAC using two separate keys?

Answers could focus for instance on properties of the algorithm that make it more vulnerable to side channel attacks than other algorithms.

Please do not only point out the fact that the nonce is fundamental to the security of the authentication tag.
–
Maarten BodewesOct 4 '13 at 22:14

2

My primary concern is with implementation aspects, where a mistake can allow message forgeries. Also 128-bit polynomial multiplication can be tricky on some platforms or programming languages.
–
Richie FrameOct 5 '13 at 6:31

My main problem with GCM is that it is not available everywhere, and hard to implement on some platforms I wish to support, like JavaCard Smart Cards.
–
fgrieuOct 5 '13 at 11:01

@fgrieu Yes, the reliance on "BigNum" multiplication is certainly a drawback. As you know I am rather heavily involved with Java Card. Normally the montgommery multiplier is not directly available. Good point (also made by Richie Frame).
–
Maarten BodewesOct 5 '13 at 17:41

2 Answers
2

In comparison against CBC mode and HMAC, GCM mode is quite commonly better alternative. But, I'll go to detail where it neccessarily is not. Just like Richie Frame, I also do not agree that CBC + HMAC is always the best comparison target. I've added few other details. Hope you find them useful.

Against CBC and HMAC

I'll discuss downsides first.

The authentication part of GCM (GHASH) is weaker than HMAC, GHASH provides maximum 128-bit authentication tag, where as HMAC allows lot longer tags (HMAC-SHA-256 would allow 256-bit authentication tag). In addition, forgery of GHASH tags in some cases is easier than HMAC:

As with any tag-based authentication mechanism, if the adversary chooses a $t$-bit tag at random, it is expected to be correct for given data with probability $1/2^t$. With GCM, however, an adversary can choose tags that increase this probability, proportional to the total length of the ciphertext and AAD. (Borrowed from NIST SP 800-38D.)

Implementations using GCM mode often uses short IV (96-bits), in fact e.g. NIST SP 800-38D recommends to use the 96-bit IVs. (Longer IVs are defined but they'll require extra invocation of GHASH function.)
The 96-bit IVs (GCM) can be to short uses where 128-bit IV (CBC) would be sufficient. Consider e.g. collision probability of random IVs.

Benefits of GCM: Properly implemented GCM is almost always faster, and it is easier to use GCM than operate AES + HMAC combination correctly. CBC mode requires padding input to the block size, thus GCM mode produces smaller output if input is not multiple of block size.

Against CTR + HMAC

Because the AES parallelization benefits are available via CTR mode as well, CTR + HMAC is fairly similar to GCM, and can perform pretty well. It can be better, in case authentication tag requirements exceed what GCM can provide.

Against CCM

The CCM mode is largely similar to GCM mode. It's largest benefit is that implementation requires less code (SW) or gates (HW), but CCM implementations generally are slower than GCM.

SIV-AES and AES-KW

It is possible to wrap other keys using AES-GCM (key wrapping).

For authenticated encryption where target is key wrapping (encryption of key with another key), SIV-AES and AES-KW algorithms are commonly better than AES-GCM. Better in the sense that they can be used without reliance to random
number generation (or a deterministic IV generator) and that their output can be smaller because there is no need IV in addition to encrypted output and authentication tag. (These are commonly considered to be benefits of deterministic authenticated encryption.)

Key, IV pair reuse

I've already mentioned IV, key pair reuse.

GCM mode does not resist reuse of Key IV combination. To prevent this, some parties like NIST put more requirements on proper use of GCM mode than the other modes. This is good idea from security point of view, but it may reduce allowed uses of the mode.

Generally, GCM is most convenient for uses where new keys are agreed or transported as a part of the protocol (TLS or IPsec). If there is use where a persistently stored key is needed, then it can be cumbersome to ensure IV uniqueness.

NIST is aware of this pitfall. NIST's CMVP (Cryptographic Module Validation Program aka FIPS 140-2 validation program) has requirement for careful use of GCM (FIPS 140-2 IG A.5). FIPS-approved implementations of GCM mode need to be very careful in their use of keys, because in GCM mode reuse of key,IV pair is not allowed to happen.

CBC-ESSIV

AES-CBC mode can be used for disk encryption (especially with ESSIV, see Wikipedia disk-encryption theory). The fact that GCM often uses small IV length and is not IV collision resistant makes it hard to apply for contexts where it is needed to encrypt a large number of blocks and encryption does not happen in-order (deterministic IV increasing would be ok). For this reason, AES-GCM is not that much used for encryption of stored data, but rather data at transit.

Summary

GCM is very good mode of operation, and it often is more convenient than legacy algorithms combinations like CBC + HMAC. However, it is not universal solution (none of the modes is), but rather it is most convenient only for large number of data at transit.

There exists a case where developers implemented a new version of the GHASH algorithm that used the new PCLMULQDQ instruction found in Intel processors, and a bug in the implementation allowed message forgery. The code change appeared to improve the performance of AES-GCM on newer processors as well as processors with additional cores that do not support PCLMULQDQ. The bug was not specific to the new instruction, but rather the new aggregation mode that allows parallel encryption of 8 blocks in order to reduce latency of hardware AES instructions on multicore processors.

The code was for OpenSSL, and occurred under the following conditions:

$({len/16})$ $mod$ $4 = 1$
$len > 64$

where $len$ is the size of the GHASHED buffer in bytes.
Values of $len$ that satisfy this condition are 80, 144, and any other value that increments from 80 by a multiple of 64. The NIST test values they used only included 64 byte messages and up to 20 bytes of additional data. The final test with no ciphertext authenticated 128 bytes of additional data. None of these meet the criteria for the bug to manifest. The bug almost made it into OpenSSL 1.0.2 but was caught in a month and fixed immediately.

The bug proceeded to send $N+1$ blocks of data to GHASH instead of $N$.. The data for the additional block $J$ was whatever happened to be in that memory location.

OpenSSL was supposed to perform the following computation to generate a new tag $T2$ from the current tag $T$ and additional message $M$ comprised of $N$ blocks (5 blocks or 80 bytes for example):

In addition to an invalid authentication tag, it had pulled 16 bytes of data from a memory location that it was not supposed to. If $J$ contained null bytes, the invalid authentication tag was equal to $T2 * H^1$

The types of attacks that can be leveraged against this bug include:

A message is sent with the incorrect implementation to a receiver
with the correct implementation, the message is forged but the
authentication tag is valid to the receiver.

A message is sent with the correct implementation to a receiver with
the incorrect implementation, the message is forged but the
authentication tag is valid to the receiver.

In the first case, the attacker must know the value of datablock J, and the first block of the additionally authenticated data must be zero. In the second case the attacker must be able to modify the incoming memory buffer.

Since the length of the ciphertext and the authenticated data are the first inputs to the GHASH function, the sizes of those are not modifiable as part of the attack. Additionally, this is more of an application mistake, rather than an issue with the algorithm itself, however the type of mistake seen here is from trying to exploit the performance advantage of GCM over CBC + HMAC. CTR + HMAC is also a very valid mode and can leverage multicore processing better without the complexity of the polynomial MAC.

The difficulty of implementing an attack on this type of bug is also quite high, as it requires attackers to have access to system memory. This is most likely to be exploited by nation states with well funded intelligence agencies. Due to the prevalence of OpenSSL, if this had made it into the wild, it would have been caught very quickly and not had much of an impact over the long term. This type of attack is valid for any polynomial MAC that operates in the same manner as GCM.

Apart from this, the original proofs that GCM bases its security on were found to be invalid in 2012, but new proofs allowed GCM to maintain its expected security.

The other primary attack vector on GCM is when short (32-bits for example) authentication tags are used, and there is enough data to reduce the security level to one which is attackable.

The best example is a VOIP application with 8-bit tags. It takes on average 128 attempts to forge a message and this allows recovery of 7-bits of $H$. Less than 1000 forgery attempts will reveal $H$ in its entirety, allowing forgeries of all packets. With an 8-bit tag, this attack is successful in under 5 minutes (speeding up the process will disrupt the audio to a point where parties may be suspicious). The expected 128:1 or better probability of forging a packet has now reached 1:1.

I should note that I was barely conscious while writing this, and there may be grammar mistakes
–
Richie FrameOct 6 '13 at 14:41

I'm not so impressed with the implementation bug. I found an error in the utility functions of SJCP that completely skipped the AAD for both the signing and verification in CCM mode. Always check your boundaries. It's interesting to read that it would lead to message forgery attacks though. I do agree that GCM spec. does allow for non-sensical tag sizes for sure. It's a bit too flexible in that sense.
–
Maarten BodewesOct 6 '13 at 14:49

Agreed about tag sizes, the $t$ - $log_2(N)$ security issue is a practical concern when t is small and N is large. The ability to reconstruct $H$ during forgery attempts is of the greatest concern, but that has been known for a while and hopefully people are aware of it.
–
Richie FrameOct 6 '13 at 14:55

That kind of bug can happen with pretty much any kind of MAC. They're the reason why I like unit tests that cover all possible message length from 0 to a few hundred.
–
CodesInChaosOct 6 '13 at 17:03