When selecting the biggest files on some DVD (such as Star Trek Zero/X) as source Handbrake does it scanning and then just stops after few seconds like it does not detect any streams and volume(?)

You do not really need to select single files as source, you select the video DVD and HandBrake should do the rest. Once the video DVD is scanned, you can select the title you want to rip and transcode.
Have you VLC Player installed?