To the best of my knowledge, the best reason to want EFI driver support in rEFInd is to provide access to filesystems. Although EFI filesystem driver choices are currently somewhat limited, those that are available can help to improve your installation and configuration options, particularly if you've found yourself "boxed in" by awkward installation or bugs, such as the dinky ESP that Ubuntu creates by default or the bug that prevents a Linux kernel with EFI stub loader support from booting from the ESP of at least some Macs.

I suppose that one of trivial reasons for this bug can be that the kernel image file (or in fact any other file) has different physical representation under FAT and other filesystems, so buggy UEFI firmware has a lot of space for generating various errors, starting from wrong alignment of the image data in the memory (what will cause EFI loader to crash), ending up with wrong calculation of the file checksum -> SecureBoot will fail.

To the best of my knowledge, the best reason to want EFI driver support in rEFInd is to provide access to filesystems. Although EFI filesystem driver choices are currently somewhat limited, those that are available can help to improve your installation and configuration options, particularly if you've found yourself "boxed in" by awkward installation or bugs, such as the dinky ESP that Ubuntu creates by default or the bug that prevents a Linux kernel with EFI stub loader support from booting from the ESP of at least some Macs.

I read somewhere that Macs don't use the ESP for booting. The ESP is there but used as a staging area for firmware updates instead. So it's not a bug, but Apple's alternative implementation of UEFI, I guess.

There's no such thing as an "alternative" implementation - the standard is clear. If Apple would implement "a bad/unexpected behaviour", then it's simply no longer a "secure boot" - it's "something else" - an unspecified proprietary "solution"...