If you want a free solution that lets you read on-the-fly from the DVD like you were trying to do with Handbrake + VLC, you might try FairMount.

This still requires VLC, but it may make Handbrake happy.
It mounts the disk as if it is already decrypted so Handbrake should be able to read the VideoTS files directly from the FairMount-mounted copy.

The solution to fix this issue can be found here [forum.handbrake.fr]. Basically download the installer for libdvdcss and install it. Then Handbrake works as before. Handbrake will not work with VLC 2.0.

I'm looking for help on the same issue. I did find a copy of libdvdcss. Based on what I've read, it sounds like that's what I need to install....but I can't install it where Handbrake is. I think my issue is that I installed handbrake incorrectly. Even though I dragged the Handbrake icon to the Applications folder, Handbrake is showing up on my desktop and in Finder as a separate Device. Does anyone out there know what I did wrong to make it show up that way? Thanks.