Edited Nov. 17th, 2018: There is now a hardware accelerated version of VLC 3.0 available from the Foundation repository. No need to compile anything anymore! The new version supports MMAL HW acceleration both in overlay mode (MMAL) and inside the video window (MMAL X11 splitter). You can use apt-get to install it.
If you have installed a self-compiled version, you must remove it first by running
sudo make uninstall
from inside your vlc directory (used for compiling)

Edited Dec. 17th, 2017: Updated to version 2.2.8 and added a few missing dependencies.
If you had installed a previous version following this tutorial, you should uninstall it first.

VLC is a great media player but the version from the Raspbian repository does not support the GPU of the RPi and is not able to run high resolution videos. We have to compile our own version to add hardware accelerated video.

Note: This version will display video in an overlay, similar to omxplayer, and not within a window. If you need this you might prefer my omxplayerGUI.

We will have to add quite a number of libraries and tools. Before doing this, you should update your system:

If you have already installed the VLC version from the repository before you must remove it now and that means every package belonging to it. A default VLC installation will install the following packages:

This will take a few minutes and if it doesn't break with some error message, we are ready for compilation. But before you can go on, you should close as many applications as possible, which includes the web browser showing this tutorial (especially if it is chromium). We will run the compilation on 3 cores and it will need a lot of free memory.

Note: Due to the way the "Raspberry Pi Desktop" (modifications) handle default desktop files, VLC will not appear immediately within the start menu (it usually needs a reboot). For a first test you can start it from the terminal by just typing "vlc".

After starting VLC for the first time, change the following options:
Settings, simple, Audio: make sure, ALSA and bcm2835 are selected
Settings, simple, Video, output: select "OpenMAX IL video output"
VLC will remember these settings.

Video is displayed in an overlay, similar to omxplayer, not inside a window. While playing, you can only control it from the keyboard.

On both RPi2 and 3 VLC will play up to 1080p30 video. It did not play some 1080p60 videos which are playing well with omxplayer.

If audio is jerky and out of sync check, if pulseaudio is installed on your system and remove or disable it.

SD video and some 720p video (RPi3) can also be played using software codecs, if you select SDL video output.

Do not try to install the repository version without uninstalling the compiled version first!

Important note: I do not consider this version (and older ones) completely stable. It definitely crashes when I stop playing recorded TS streams typing "s". Live TV stream are playing well (same format).

Thanks to cjan who discovered the patch and showed how to apply it.

Last edited by gkreidl on Sat Nov 17, 2018 6:45 am, edited 4 times in total.

Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

"We will now set and export some environment variables for configuration and compilation"

If I remove "-mcpu=cortex-a7 -mfpu=neon-vfpv4" from the environment variables, will the ./configure still pull the correct information from my system? I was trying to shortcut the process because I was compiling VLC for my RPi Zero W and removed those two. It compiled and installed successfully, but when I play a DVD it stutters slightly and I was curious if it was because I didn't indicate the mcpu and mfpu which made it un-optimized. The only reason I'm suspicious is because it will play 1080p files off of a USB but can't play a DVD. Anyone out there with success stories for RPi Zero W?

"We will now set and export some environment variables for configuration and compilation"

If I remove "-mcpu=cortex-a7 -mfpu=neon-vfpv4" from the environment variables, will the ./configure still pull the correct information from my system? I was trying to shortcut the process because I was compiling VLC for my RPi Zero W and removed those two. It compiled and installed successfully, but when I play a DVD it stutters slightly and I was curious if it was because I didn't indicate the mcpu and mfpu which made it un-optimized. The only reason I'm suspicious is because it will play 1080p files off of a USB but can't play a DVD. Anyone out there with success stories for RPi Zero W?

I don't recommend using VLC (even HW accelerated) on a first generation RPi or Zero. Long time ago I created a version for them (and published a tutorial), but it didn't work reliably for full HD videos, for example. The software overhead is much bigger than with omxplayer.

Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

OK. Well I finally got it working by reinstalling my OS, running through the tutorial again, and doing about a dozen apt-get install xxxxx commands on all the libraries that didn't seem to get installed with this command for some reason:

And it now works smooth and fullscreen on the pi via the HDMI output. However, it's still incredibly choppy and jittery when streaming out over http. Is there a particular codec that should be used with this hardware acceleration method? I've tried h264, mp4a, mp1a, mp2a, etc with almost no difference in quality.

I'm happy to start a new thread for this, it just seems like people who need this tutorial might also have similar problems, so wanted to keep it here for later reference.

OK. Well I finally got it working by reinstalling my OS, running through the tutorial again, and doing about a dozen apt-get install xxxxx commands on all the libraries that didn't seem to get installed with this command for some reason:

And it now works smooth and fullscreen on the pi via the HDMI output. However, it's still incredibly choppy and jittery when streaming out over http. Is there a particular codec that should be used with this hardware acceleration method? I've tried h264, mp4a, mp1a, mp2a, etc with almost no difference in quality.

I'm happy to start a new thread for this, it just seems like people who need this tutorial might also have similar problems, so wanted to keep it here for later reference.

Thanks.

-mike

HW acceleration in VLC works only for screen display, not for streaming.

Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

The make step [make -j3] doens't take long and returns the following error:

libtool: error: 'meta_engine/libtaglib_plugin_la-taglib.lo' is not a valid libtool object

I can't find an answer elsewhere to this, so perhaps the issue is known here? I executed all statements exactly as posted here.

Also, I want to express my appreciation for this guide, since for a newbie installing programs seems very different [and more difficult!] when compared to OSX of Windows.

It's hard to tell what went wrong. Run all the "apt-get" lines again, then enter the vlc folder again and start again with the "export ..." line and run all following commands again; watch the output of the configure command.

BTW, installing software (e. g. VLC) from a repository is really simple. But in this case we want to compile and install a VLC version from source with special options, which are not supported by the repository version.

Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

libavcodec57 is already the newest version (7:3.2.8-1~deb9u1).
libavcodec57 set to manually installed.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
libavcodec-extra57 : Conflicts: libavcodec57 but 7:3.2.8-1~deb9u1 is to be installed
E: Unable to correct problems, you have held broken packages.

EDIT:
Removed/reinstalled libavcodec57 and got rid of the dependency error, but still had the no package 'libavcodec' error from configure. It's there according to dpkg (version problem?). Ran configure with the --disable-avcodec switch just for fun and was able to compile and run the vlc gui, but when pIaying a video I get:

libavcodec57 is already the newest version (7:3.2.8-1~deb9u1).
libavcodec57 set to manually installed.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
libavcodec-extra57 : Conflicts: libavcodec57 but 7:3.2.8-1~deb9u1 is to be installed
E: Unable to correct problems, you have held broken packages.

EDIT:
Removed/reinstalled libavcodec57 and got rid of the dependency error, but still had the no package 'libavcodec' error from configure. It's there according to dpkg (version problem?). Ran configure with the --disable-avcodec switch just for fun and was able to compile and run the vlc gui, but when pIaying a video I get:

For what it's worth, I hope you don't give up. I compiled VLC for my Pi Zero W which takes significantly longer in every respect; so much so that I had to start "sudo make" in the evening just so it would finish before I left for work the next morning. It's rather satisfying playing videos from my Zero W now though...
I got a similar error "No available XVideo adaptor" when trying to compile MPV; something to do with OpenGL support I think but it has been some time since I did that.