In a nutshell, you can retain VLC playback, with menus, by sticking with VLC 3.0.4. Or, you can turn OFF SIP on your Mac and use the latest version available of VLC. OR, you can install High Sierra on your Mac, turn ON SIP, and use any version of VLC you want.

Unless you can find a hacked version of later VLC that does not have runtime hardening enabled.

Here is my post explaining the issue from the Mac OS MakeMKV forum:

I looked up the changelog for VLC - saw the following:

macOS:

* Starting with VLC 3.0.5, VLC will be distributed with runtime hardening
enabled on macOS Mojave.
All external VLC plugins need to be signed by a DeveloperID certificate in order
to continue working with the official VLC package.

Did a little bit of reading on that, and found this applies to unsigned .dylibs. Just on a hunch, reinstalled VLC 3.0.6, turned off SIP on my Mojave (10.14.2) Mac, and tried playing both the BluRay that I got the missing AACS error on, and another randomly chosen one. Both played fine, with menus, good playback. Plenty of instructions available on turning off SIP, so I won't go into that. 2 choices - turn it off and use the newest VLC, or stick with 3.0.4. UNLESS there is a way to find a modified (unofficial) VLC package.

After reading my previous post, it seems clear that I was half-asleep when I wrote it. I figured it out, so please allow me to share my procedure.

I attempted to use the method written by @Billycar11 earlier in this thread to get VLC to play a Blu-ray disc. Unfortunately, this wasn't an automatic win. I think the disconnect is the 32-bit versus 64-bit libraries. Instead of copying "libmmbd.dll" from the MakeMKV directory, you nee to copy the "libmmbd64.dll" file (as detailed in the procedure below).

Copy the libmmbd64.dll file from the 'C:\Program Files (x86)\MakeMKV\' directory and made two duplicates in the 'C:\Program Files\VideoLAN\VLC' directory.

Move the 'libaacs.dll' and 'libbdplus.dll' files out of the 'C:\Program Files\VideoLAN\VLC' directory. You can just create another subdirectory (e.g. "C:\Program Files\VideoLAN\VLC\Archive") and drop them in there.

Rename the two "libmmbd64.dll" duplicates to "libaacs.dll" and "libbdplus.dll" (respectively).

Fire-up the VLC, load a Blu-ray, and enjoy!

MakeMKV v1.14.3 for Windows is both x86 and x64 compatible, so it has libraries for both architectures.

This worked for me to, but I didn't initially have the 'libaacs.dll' and 'libbdplus.dll' files in my VLC directory. Creating the copies from MakeMKV did the trick!