The need

particular video decompression algorithm like stk11xx or usbvision (the decompression algorithm is in kernelspace since it is of linear complexity but shall be moved to userspace)

colorspace translations

filters that be done in hardware if the selected hardware can, otherwise software plugin

The general constraints

Application shall still have access to the basic kernel driver. Thus, applications will have access to the 'raw' input data without software post-processing. Application developers love to develop optimized codecs :)

The original v4l driver infrastructure should not change. We do not want to break any drivers. However, the extended v4l driver will use common functionalities like video_ioctl2 if this contributes to simplify the architecture.

As a consequence, the v4l API does not change for applications.

v4l driver and extended v4l driver shall mutually exclusive since they share the same hardware

Limit as much as possible kernel memory and CPU consumption.

Elegant, flexible.This is why I specify before coding :)

A solution

I thought it would be funny to create userspace drivers like FUSE based projects do.
Applications would see nothing but a new device driver to access, all decompression algorithm would be in userspace.
The path from the application to the v4l2 kernel driver:
[Application]<->[/dev/video1 (created by FUSE-like driver)]<->[/dev/video0 (created by kernel v4l2 driver)]
For applications, this changes nothing...
For v4l drivers for hardware that can fullfill all application requested formats, this changes... nothing too.
This architecture is interesting for v4l drivers that interface hardware that provide proprietary compressed video pictures or that are not able to fullfill the application needs by hardware.

The only change for the application is only a new device to access (/dev/video1) so there is no extension of the v4l API/ABI.
The only change for the user is to launch another daemon in the starting scripts (/etc/rc.d/...). The v4l driver does not change, only extension points will be implemented if necessary.

The extended v4l driver is an instance of a v4l driver with some entry points for the helper daemon

Specification

First, some obvious assumptions:

The API available to the applications will be named v4l public API (headers in path /usr/include/linux/)

All extensions will be made by internal APIs. I will name it v4l extension private API (headers in path /usr/src/linux/drivers/media/video/...)

I will name uncompressed frames formats the ones defined in the v4l public API.

I will name compressed frames formats the ones not defined in the v4l public API.

Open/close access the hardware

Summary

This step corresponds to the first and the last access to hardware via the extended driver.
Context
The application opens the extended v4l driver. The scenario will assume that /dev/video0 is the base device entry point and /dev/video1 is the extended entry point.

Description

The application opens /dev/video1.

The extended device driver asks the base driver to lock the device.

The application controls/gets incoming frames from the hardware (see further).

The application closes /dev/video1.

The extended device driver wipes out all internal allocations and asks the v4l helper to do so...

Control the hardware

Summary

This step corresponds mainly to the ioctl() entry points' behaviour.
The extended v4l driver will have to interact with the v4l base driver and with the v4l helper daemon with specific ioctls.
The scenario will show the typical controls the application will address to the extended v4l driver.
In most cases, the extended v4l driver will ask the base v4l driver to do the job. The v4l helper daemon will be used on specific control steps.

Context

The application needs to get frames from the v4l peripheral, this task is fulfilled by the extended v4l driver with the help of the base v4l driver.
Pre-conditions

The extended driver is opened properly on /dev/video1.

Description

The application gets the device hardware capabilities with VIDIOC_QUERYCAP.

For all these previous steps, the extended video driver transmits the request to the base driver without interaction.

The application negociates with the extended v4l driver the video data format, with VIDIOC_ENUM_FMT, VIDIOC_S_FMT.

If the v4l helper is here, see the extension point named "Provide compatible video formats". Otherwise, see alternative A1

Alternatives

A1 - the v4l helper daemon is not running (the /dev/v4lctl driver is not opened by the helper daemon)
the extended v4l driver transmits the request to the base driver without interaction...

Get video frames

Summary

As the driver is correctly configured, the hardware captures and streams data from the device.
This step will show how to managed the incoming data and provide them to the Application.
This is the most important part to understand from this specification.

Pre-conditions

The extended v4l driver is properly configured on /dev/video1.

Description

The application requests the driver to allocate frames with VIDIOC_REQBUFS.

The application requests each allocated buffer characteristics with VIDIOC_QUERYBUF.

The application requests the driver to enqueue each frame with VIDIOC_QBUF.

The application starts the acquisition chain with VIDIOC_STREAMON.

The application waits and dequeues a frame with VIDIOC_DQBUF.

The extended v4l driver waits and dequeues a frame from the base v4l driver. If the frame needs to be uncompressed, see the extension point named "Uncompress frame".

Otherwise the extended v4l driver transmits the request to the base driver without interaction.

As soon as the frame is processed, the application calls VIDIOC_QBUF to re-enqueue this buffer.

The VIDIOC_DQBUF/QBUF mechanism is repeated to get each frame sequentially.

The application stops the acquisition with VIDIOC_STREAMOFF.

Exceptions

EX1 The v4l helper daemon is not active or does not respond correctly to the extended v4l device driver request
The extended v4l device driver returns -EFAULT (internal error) to the application, this ends the process.

Provide compatible video data formats

Summary

This step specifies how the extended v4l driver is able to provide to the application the hardware and software compatible video data formats

Pre-conditions

The v4l helper daemon has been registered

Description

Either the application has requested to enumerate all supported video formats,

The extended v4l driver ask the v4l helper daemon all possible formats for this hardware.

The v4l helper daemon provides all additional software video format.

Or the applications has requested to set a specific video format,

The extended v4l driver asks the v4l helper daemon if the requested video format is possible for this hardware

If the helper daemon is able to process by software the video format, it acknowledges with the negociated video format. Otherwise it returns a NACK

Uncompress frames

Summary

This part concerns the interactions with the userland v4l helper in order to uncompress frames.

Context

In order to precise the context of a frame decompression, here is a collaboration overview of all components involved.

Pre-conditions

The extended v4l driver is properly configured on /dev/video1.
The v4l helper is present and available (has answered correctly to all steps before)
The extended v4l driver received a new compressed frame from the base v4l driver

Description

The extended v4l driver prepares data for the v4l helper daemon. This implies some generic preprocessing such as eventual memcopies.

The extended v4l driver asks the v4l helper daemon to uncompress the newly available frame