News

2010-06-22 V4L mini-summit on June, 14-16 2010

A V4L mini-summit was held in Nokia Research Center, in Helsinki, during this
Summer, with more than 20 participants.

With a massive participation of V4L developers from several parts of the world,
the Helsinki's mini-summit just ended. Several interesting topics were discussed
there, including videobuf, v4l1, test/compliance applications, libv4l, etc.

The main focus of the summit was how to proceed in the next several months
with regards to the SoC support in V4L. Video on SoCs is big business and
Google's Android platform plays a major role in pushing linux into the mobile
area. SoCs have unique requirements and for the past few years the foundation
has been laid in the V4L subsystem to start supporting the full functionality
of that hardware. Much of the summit was about what is still needed to achieve
this.

Part of the work is of course to introduce the new APIs and functionality that
are needed, but another part of the work is to figure out what to do with legacy
functionality that is actually blocking or hampering these new developments.
Sadly, throughout the V4L history not all decisions and designs where always
thought out carefully and so we have some hacks and inconsistencies that need
to be resolved.

Obviously not everything could be decided in just three days: some of the
items need more discussions.

I would like once again to thank all participants for their constructive input
and enthusiasm for this project. It is great to see so much interest in this.

I will go through the agenda points one-by-one. Any mistakes there may be are
mine and mine alone. Please reply to correct any errors!

1) Presentation on the Qualcomm video hw architecture.

Jeff Zhong gave the presentation. The main restriction that they have is that
they have a lot of proprietary video optimization and processing code and that
must somehow sit between the application and the driver.

A good solution was found in the form of a libv4l plug-in that uses IPC to talk
to a separate daemon process. Due to legal concerns it was not possible to
eliminate the daemon and just put everything in the plug-in code. See also
item 12.

2) Presentation on the ST-Ericsson video hardware.

Robert Fekete gave the presentation. This link gives an overview of
their hardware.

They have several memory-to-memory devices, so they were very interested in
that recently merged functionality.

An open issue is how to create a blitter driver. The suggestion was to discuss
this further with Samsung and Andrew Morton.

For HDMI receivers and transmitters the foundations are already in place (high
resolution API and events API). For CEC support in HDMI more research and
discussions are needed: should this be an input driver? Or part of the IR
framework?

They also pointed to the openWF streaming API that is being developed. See
khronos.org.
This will be of interest for future videobuf discussions.

3) Removal of V4L1: status of driver conversion in the kernel, status of
moving v4l1->v4l2 conversion into libv4l1.

V4L1 driver status:

se401:

Hans de Goede will convert this to V4L2 (gspca)

ibmcam + konicawc:

Hans de Goede will convert this to V4L2 (gspca)

stradis:

Hans Verkuil will contact the company whether they have any interest.
Firmware is needed to test these cards, but seems to be unavailable.
No work has been done on this driver for a long time.
If there is no interest, then this driver will be removed.

pwc:

Hans Verkuil removed the V4L1 support, but that broke the driver.
It is now possible to test the hardware, so this will need a second
attempt.

Was converted, Alan Cox has the hardware, but has not yet tested the
patch. If remains untested, then just merge anyway.

Those drivers that will be removed will be marked deprecated and put on the
feature removal list for 2.6.35. Actual removal will take place in 2.6.37.

libv4l1 status:

All V4L1 ioctls are now implemented by the library, but it needs testing
before it can be released.

This means that the V4L1 compat layer in the kernel can be marked deprecated
for 2.6.35 and put on the feature removal list so we can remove it in 2.6.37.
This will be a great improvement since the V4L1 compat support in
videobuf-dma-sg.c is really blocking videobuf development.

So V4L1 will finally disappear from the kernel in 2.6.37! There should be
an LWN article as well so everyone (we hope) will be informed of this in time.

4) Adopting old V4L1 programs

Hans de Goede suggested that we pull in the unmaintained camorama tool. No
other programs came up.

5) Media Controller Roadmap

Presentation by Laurent Pinchart.

It was suggested that there should be a MC ioctl that returns version information
of both hardware and the MC API. Everyone agreed on this, so this will be added.

There is an open issue on how to configure all the links in a complex video system.
In the current prototype implementation they become active when streaming starts
and cannot be changed while streaming is in progress. But in other systems
something like a commit ioctl might be needed or some other way to set multiple
links atomically.

Another idea was to have presets for common routing configurations and be able
to enumerate those presets. Whether that should be in user space or kernel space
was unclear, though.

Similar problems occur with configuring a large complex video subsystem. Perhaps
some configuration snapshot functionality might come in handy here.

These last three points will all need more thought.

The roadmap for the Media Controller is that the first patches should appear in
1-2 weeks. First adding subdev device nodes, then the MC itself.

6) TO DO list regarding V4L2 core framework including the new control framework.

Control Framework: done, just needs a pull request. Some discussions on how
to handle 'auto-foo' and 'foo' controls (e.g. autogain/gain). Can be fixed
later.

Replace all s/g/enum/try_mbus_fmt subdev ops with s/g/try_busfmt. Mostly done, want to
finish this for 2.6.36.

Handling of dbg_g/s/_register and dbg_g_chip_ident can be moved into the core.
Should simplify subdev drivers.

It was decided not to post the control framework yet, Laurent Pinchart will
attempt to use it in UVC first. If that succeeds, then Hans Verkuil can post a
pull request for this.

There was some discussion on how to reduce the amount of printk's in drivers.
Possibly by somehow using debugfs. No concrete results, though.

We should have some well-defined projects for Google Summer of Code (or similar
initiatives). Ideas are: an RDS test program, a dummy libv4l plugin.

One other useful thing to add to the TODO list is to move the pixel format
descriptions into the core framework to give consistent namings.

7) soc-camera status

Presentation by Guennadi Liakhovetski.

There are basically three outstanding issues when it comes to the remaining
soc-camera dependencies in sensor drivers:

Bus parameter configuration. This should be done through a board info config.
Needs new standard structs for basic parameters: one for parallel sensors,
one for MIPI sensors. These can be embedded in the board info config.

Control handling. Will be easy to fix once the control framework is merged.

Mediabus pixel code defines: a discussion was started some time ago on the
mailinglist regarding some ambiguities of the YUV pixel codes. This discussion
needs to be finished so we can come to a conclusion. Since these defines will
become public in the near future this issue needs to be resolved soon.

8) V4L2 video output vs. framebuffer.

Guennadi's idea was to provide a generic framebuffer driver that could sit
on top of a v4l output driver.

After some discussion it became clear that there was not enough interest in
this. Creating a fb driver is already very simple and the general opinion was
that there is no need for something like this at the moment.

That said, the framebuffer subsystem sees very little effort in improving the
APIs, in part because there is no central maintainer of that subsystem anymore.

Many SoCs support the fb API though, so the slow progress in the fb subsystem
can become a problem.

Samsung: will soon post subdev-based framebuffer driver to the list. (Already
done: see the patch series from Sylwester Nawrocki).

There are some common FB operations that most SoC vendors need. Samsung will
look at that and try to make a proposal for such common ioctls and post them
to the list, with a CC to Andrew Morton (due to the unmaintained status of the
FB API). See also item 15.

9) Remote Controllers.

Presentation by Mauro Carvalho Chehab.

One question that needed answering was whether RC keymaps should (optionally)
remain in kernel space, or whether all could be moved to userspace. Nobody
could see a good use case for keeping this in the kernel and the consensus
was to move this all to userspace.

As mentioned before, CEC was discussed in this context as well. More research
needs to be done to see whether the IR framework is a good match for this.

One currently missing bit in the IR framework are IR transmitters. Work is
being done on this. The implementation will use 'raw IR' format. Should CEC
use this framework as well, then CEC will need a non-raw format. This is
something that needs to be discussed further.

10) V4L2 driver compliance test framework.

Hans Verkuil made a start with this with the v4l2-compliance utility. This is
currently very basic and limited, though. Help is needed to extend this tool
so that it can cover the whole V4L2 API.

There is a sourceforge project called v4l-test. We have permission from the
author to reuse code from that project in v4l2-compliance. Some companies also
have in-house test tools. So there is a lot out there, we just need someone
to actually put in the work to turn v4l2-compliance into a really useful tool.

Some offers of help were made, so hopefully we can see more progress on this
in the near future.

Another idea was to test the other way around: verify user programs through
vivi and/or a libv4l plugin and/or a new driver. Not everyone liked to have
a new driver for this, though.

A general problem is what should be done with reserved fields that userspace
should zero. Currently there is no check whether apps really zeroed them.
So if you start to use them later, then apps might unexpectedly fail.

The eventual proposal was to have extra checks in the core, using printk to
warn that reserved fields are not zeroed by the app. Possibly only active if
the ADVANCED_DEBUG config option is enabled.

It was also suggested to get rid of the ADVANCED_DEBUG config option and use
debugfs instead.

10a) V4L2 specification changes/clarifications

The previous discussion led to a new discussion regarding ambiguities in the
V4L2 spec:

Hans de Goede: should we allow STREAMON if no buffers are queued? This should
be stated clearly in the spec. Discuss on mailinglist which option to use.

Mixed read() and mmap() streaming. Allow or disallow? bttv allows it, which
is against the spec since it only has one buffer queue so a read() will steal
a frame. No conclusion was reached. Everyone thought it was very ugly but
some apps apparently use this. Even though few drivers actually support this
functionality.

Hans de Goede: Clarify read/write call in spec. JPEG writing: write must
always do full JPEG frames as partial JPEG frames make no sense (would require
that the driver detects the end of the JPEG frame!).
JPEG readining: libv4l emulates read by giving you whatever fits of the frame
and throws away the rest of the frame, while many drivers allow you to read
frames piecemeal, especially when it comes to raw or MPEG formats. But does
that make sense for JPEG-like formats?
To be discussed further on the mailinglist.

Hans Verkuil: Remove this 'feature' from the spec: 'Regardless if /dev/video
(81, 0) or /dev/vbi (81, 224) is opened the application can select any one of
the video capturing, overlay or VBI capturing functions.' No drivers and apps
do this and it is extremely complex to implement anyway. The consensus was
that this could be removed.

11) Discuss list of 'reference' programs to test against.

A discussion was started what to do with the 'overlay' support (PCI-to-PCI)
in V4L2. Currently only supported by xawtv3 and xdtv, but no longer works
out of the box: you need to manually setup the framebuffer address in the device
node.

Some where in favor of killing this, others thought it still had merits in
some cases. To be discussed on the mailinglist.

12) A processing plugin API for libv4l.

The ability to create proprietary plug-ins is very important for SoC vendors.

The in-depth discussion of the proposal happened in a separate track. I was
not present, so I can only report the outcome:

The initial RFC was found to be too limited. Instead there should be a
generic plug-in API to capture all ioctls in both directions.

This makes it even possibly to create a completely emulated device (although
that won't show up in sysfs, obviously).

A new RFC will be prepared and posted by Hans de Goede.

13) Meego presentation

Sakari Ailus gave a presentation on the camera framework in Meego. They used
to have a daemon to handle proprietary software but will move to a libv4l
plug-in instead.

They have a 'v4l2camsrc' userspace module that contains support for custom
control IDs. Sakari Ailus will look into making a proposal to add those controls
to the standard V4L framework. Many of those controls seemed suitable as
standard controls. Having this in the core would make it possible to ditch this
module.

14) Status of intel drivers.

Presented by Xiaolin Zhang.

Mostly done. The comments from the last code review are being addressed.
Two open items:

15) Status of the Texas Instruments drivers

the display subsystem (DSS) uses sysfs to address deficiencies in the FB API.
TI will work with Samsung on trying to come up with new common FB API additions.
(See also item 8).

Full support for omap3x should be finished this year.

Vaibhav's team is now also handling the DMxxx support.

16) videobuf/videobuf2.

Presentations by Laurent Pinchart and Pawel Osciak.

Everyone agreed that videobuf had too many fundamental flaws to be refactored
into a working framework. So there will be a videobuf2. Hopefully the current
videobuf drivers can be converted to videobuf2 eventually.

The recommendation was to clean up and fix the current videobuf as much as
possible before creating videobuf2.

Detailed discussions will have to wait until the videobuf2 patches are posted.
Everyone agreed with the basic concept, though. In particular having custom
memory allocators was seen as essential.

The 'waitqueue per buffer' method should be replaced with a single waitqueue
in the queue itself. This allows out-of-order dequeuing, something that is
needed by the Samsung hardware.

ST-Ericsson mentioned that the current enum v4l2_colorspace has no support
for full range vs limited range colorspaces. New colorspace entries should be
added to differentiate between the two.

The Samsung multi-planar proposal raised several issues:

bytesperline in v4l2_pix_format is not sufficient in the multiplanar case:
should we add a v4l2_pix_format_plane with an array of bytesperline?

how do you know which plane contains what? The consensus was that fourcc codes
should determine uniquely what goes to which plane.

not clear how to specify that multiplanar is needed:

new v4l2_buf_type?

new pixel fourcc formats?

new memory type format?

an alternative was also proposed: we need to be able to preregister memory
to prepare the memory for use with video streaming (e.g. cache configurations,
locking it in memory). One option is to preregister all planes. After that
you can refer to the whole set of planes through a single ID as returned
by the prereg ioctl. This might change the public API of the multiplanar
proposal substantially (the internals will remain pretty much the same,
though). Samsung and ST-Ericsson will make a joint proposal for this.

A next step will be to have a kernel-central way of creating large memory
buffers that can be used by v4l, fb, graphics (i.e. openGL textures) and
possibly others (pass on to DSPs for example).

This will need some cross-subsystem cooperation. ST-Ericsson has code and a
proposal. But this will definitely need a separate brainstorm session with
the relevant experts.

17) Ideas for LPC media track.

The Linux Plumbers Conference will be held November 3-5 in Cambridge, MA.
It will contain a media track and Mauro wanted to have some feedback which
topics should be discussed:

alsa & HDMI

alsa & v4l2 timestamping (ktime)

query v4l2 output latency

X memory handling

Can parts of kms be shared with v4l? E.g. EDID parsing, others?

Memory pool: how usable are gem, ttm?

remote controller & X & apps

18) v4l2-int-device.h removal

Currently only used by one i2c driver and the omap2 driver. Should be converted
to subdev. Sakari Ailus and Hans Verkuil will discuss offline who can do what.

This concludes the report of this summit. As has become clear, there are many
ongoing developments taking place on many different levels. We are seeing steady
progress in all areas, every kernel release has one or more new building blocks
in place. It looks like we are getting close to finally offering the functionality
that SoC drivers need so that we can start to merge them. Perhaps not yet with a
full feature set, but good enough so we can build from there.

It is also clear that libv4l is developing into a much more important library
then was originally envisaged. It's likely to become an essential library for
SoCs. I'm very pleased to see how neatly it fits into the bigger picture.

A big 'Thank You!' to everyone who helped make this mini-summit a success!