Discovering a device internal topology, and configuring it at runtime, is one
of the goals of the media framework. To achieve this, hardware devices are
modelled as an oriented graph of building blocks called entities connected
through pads.

An entity is a basic media hardware building block. It can correspond to
a large variety of logical blocks such as physical hardware devices
(CMOS sensor for instance), logical hardware devices (a building block
in a System-on-Chip image processing pipeline), DMA channels or physical
connectors.

A pad is a connection endpoint through which an entity can interact with
other entities. Data (not restricted to video) produced by an entity
flows from the entity’s output to one or more entity inputs. Pads should
not be confused with physical pins at chip boundaries.

A link is a point-to-point oriented connection between two pads, either
on the same entity or on different entities. Data flows from a source
pad to a sink pad.

A media device is represented by a struct media_device
instance, defined in include/media/media-device.h.
Allocation of the structure is handled by the media device driver, usually by
embedding the media_device instance in a larger driver-specific
structure.

Entities are represented by a struct media_entity
instance, defined in include/media/media-entity.h. The structure is usually
embedded into a higher-level structure, such as
v4l2_subdev or video_device
instances, although drivers can allocate entities directly.

Interfaces are represented by a
struct media_interface instance, defined in
include/media/media-entity.h. Currently, only one type of interface is
defined: a device node. Such interfaces are represented by a
struct media_intf_devnode.

Pads are represented by a struct media_pad instance,
defined in include/media/media-entity.h. Each entity stores its pads in
a pads array managed by the entity driver. Drivers usually embed the array in
a driver-specific structure.

Pads are identified by their entity and their 0-based index in the pads
array.

Both information are stored in the struct media_pad,
making the struct media_pad pointer the canonical way
to store and pass link references.

Links are represented by a struct media_link instance,
defined in include/media/media-entity.h. There are two types of links:

1. pad to pad links:

Associate two entities via their PADs. Each entity has a list that points
to all links originating at or targeting any of its pads.
A given link is thus stored twice, once in the source entity and once in
the target entity.

The media framework provides APIs to iterate over entities in a graph.

To iterate over all entities belonging to a media device, drivers can use
the media_device_for_each_entity macro, defined in
include/media/media-device.h.

structmedia_entity*entity;media_device_for_each_entity(entity,mdev){// entity will point to each entity in turn...}

Drivers might also need to iterate over all entities in a graph that can be
reached only through enabled links starting at a given entity. The media
framework provides a depth-first graph traversal API for that purpose.

Note

Graphs with cycles (whether directed or undirected) are NOT
supported by the graph traversal API. To prevent infinite loops, the graph
traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH,
currently defined as 16.

Due to the wide differences between drivers regarding power management
needs, the media controller does not implement power management. However,
the struct media_entity includes a use_count
field that media drivers
can use to track the number of users of every entity for power management
needs.

The media_entity.use_count field is owned by
media drivers and must not be
touched by entity drivers. Access to the field must be protected by the
media_device.graph_mutex lock.

When starting streaming, drivers must notify all entities in the pipeline to
prevent link states from being modified during streaming by calling
media_pipeline_start().

The function will mark all entities connected to the given entity through
enabled links, either directly or indirectly, as streaming.

The struct media_pipeline instance pointed to by
the pipe argument will be stored in every entity in the pipeline.
Drivers should embed the struct media_pipeline
in higher-level pipeline structures and can then access the
pipeline through the struct media_entity
pipe field.

Calls to media_pipeline_start() can be nested.
The pipeline pointer must be identical for all nested calls to the function.

media_pipeline_start() may return an error. In that case,
it will clean up any of the changes it did by itself.

Link configuration will fail with -EBUSY by default if either end of the
link is a streaming entity. Links that can be modified while streaming must
be marked with the MEDIA_LNK_FL_DYNAMIC flag.

If other operations need to be disallowed on streaming entities (such as
changing entities configuration parameters) drivers can explicitly check the
media_entity stream_count field to find out if an entity is streaming. This
operation must be done with the media_device graph_mutex held.

Link validation is performed by media_pipeline_start()
for any entity which has sink pads in the pipeline. The
media_entity.link_validate() callback is used for that
purpose. In link_validate() callback, entity driver should check
that the properties of the source pad of the connected entity and its own
sink pad match. It is up to the type of the entity (and in the end, the
properties of the hardware) what matching actually means.

Subsystems should facilitate link validation by providing subsystem specific
helper functions to provide easy access for commonly needed information, and
in the end provide a way to use driver-specific callbacks.

Drivers may register a callback to take action when new entities get
registered with the media device. This handler is intended for creating
links between existing entities and should not create entities and register
them.

Optional device driver name. If not set, calls to
MEDIA_IOC_DEVICE_INFO will return dev->driver->name.
This is needed for USB drivers for example, as otherwise
they’ll all appear as if the driver name was “usb”.

serial

Device serial number (optional)

bus_info

Unique and stable device location identifier

hw_revision

Hardware device revision

topology_version

Monotonic counter for storing the version of the graph
topology. Should be incremented each time the topology changes.

id

Unique ID used on the last registered graph object

entity_internal_idx

Unique internal entity ID used by the graph traversal
algorithms

entity_internal_idx_max

Allocated internal entity indices

entities

List of registered entities

interfaces

List of registered interfaces

pads

List of registered pads

links

List of registered links

entity_notify

List of registered entity_notify callbacks

graph_mutex

Protects access to struct media_device data

pm_count_walk

Graph walk for power state walk. Access serialised using
graph_mutex.

source_priv

Driver Private data for enable/disable source handlers

enable_source

Enable Source Handler function pointer

disable_source

Disable Source Handler function pointer

ops

Operation handler callbacks

Description

This structure represents an abstract high-level media device. It allows easy
access to entities and provides basic media device-level support. The
structure can be allocated directly or embedded in a larger structure.

The parent dev is a physical device. It must be set before registering the
media device.

model is a descriptive model name exported through sysfs. It doesn’t have to
be unique.

enable_source is a handler to find source entity for the
sink entity and activate the link between them if source
entity is free. Drivers should call this handler before
accessing the source.

disable_source is a handler to find source entity for the
sink entity and deactivate the link between them. Drivers
should call this handler to release the source.

Use-case: find tuner entity connected to the decoder
entity and check if it is available, and activate the
the link between them from enable_source and deactivate
from disable_source.

Note

Bridge driver is expected to implement and set the
handler when media_device is registered or when
bridge driver finds the media_device during probe.
Bridge driver sets source_priv with information
necessary to run enable_source and disable_source handlers.
Callers should hold graph_mutex to access and call enable_source
and disable_source handlers.

This function initializes the media device prior to its registration.
The media device initialization and registration is split in two functions
to avoid race conditions and make the media device available to user-space
before the media graph has been completed.

So drivers need to first initialize the media device, register any entity
within the media device, create pad to pad links and then finally register
the media device by calling media_device_register() as a final step.

The caller is responsible for initializing the media_device structure
before registration. The following fields of media_device must be set:

media_entity.dev must point to the parent device (usually a pci_dev,
usb_interface or platform_device instance).

media_entity.model must be filled with the device model name as a
NUL-terminated UTF-8 string. The device/model revision must not be
stored in this field.

The following fields are optional:

media_entity.serial is a unique serial number stored as a
NUL-terminated ASCII string. The field is big enough to store a GUID
in text form. If the hardware doesn’t provide a unique serial number
this field must be left empty.

media_entity.bus_info represents the location of the device in the
system as a NUL-terminated ASCII string. For PCI/PCIe devices
media_entity.bus_info must be set to “PCI:” (or “PCIe:”) followed by
the value of pci_name(). For USB devices,the usb_make_path() function
must be used. This field is used by applications to distinguish between
otherwise identical devices that don’t provide a serial number.

media_entity.hw_revision is the hardware device revision in a
driver-specific format. When possible the revision should be formatted
with the KERNEL_VERSION() macro.

Note

Upon successful registration a character device named media[0-9]+ is created. The device major and minor numbers are dynamic. The model name is exported as a sysfs attribute.

Entities are identified by a unique positive integer ID. The media
controller framework will such ID automatically. IDs are not guaranteed
to be contiguous, and the ID number can change on newer Kernel versions.
So, neither the driver nor userspace should hardcode ID numbers to refer
to the entities, but, instead, use the framework to find the ID, when
needed.

The media_entity name, type and flags fields should be initialized before
calling media_device_register_entity(). Entities embedded in higher-level
standard structures can have some of those fields set by the higher-level
framework.

The registration code assigns minor numbers and registers the new device node
with the kernel. An error is returned if no free minor number can be found,
or if the registration of the device node fails.

Zero is returned on success.

Note that if the media_devnode_register call fails, the release() callback of
the media_devnode structure is not called, so the caller is responsible for
freeing any data.

Media entity objects are often not instantiated directly, but the media
entity structure is inherited by (through embedding) other subsystem-specific
structures. The media entity type identifies the type of the subclass
structure that implements a media entity instance.

This allows runtime type identification of media entities and safe casting to
the correct object type. For instance, a media entity structure instance
embedded in a v4l2_subdev structure instance will have the type
MEDIA_ENTITY_TYPE_V4L2_SUBDEV and can safely be cast to a v4l2_subdev
structure using the container_of() macro.

An unique internal entity specific number. The numbers are
re-used if entities are unregistered or registered again.

pads

Pads array with the size defined by num_pads.

links

List of data links.

ops

Entity operations.

stream_count

Stream count for the entity.

use_count

Use count for the entity.

pipe

Pipeline this entity belongs to.

info

Union with devnode information. Kept just for backward
compatibility.

Description

Note

stream_count and use_count reference counts must never be
negative, but are signed integers on purpose: a simple WARN_ON(<0)
check can be used to detect reference count bugs that would make them
negative.

This routine initializes the embedded struct media_gobj inside a
media graph object. It is called automatically if media_*_create
function calls are used. However, if the object (entity, link, pad,
interface) is embedded on some other object, this function should be
called before registering the object at the media controller.

As the number of pads is known in advance, the pads array is not allocated
dynamically but is managed by the entity driver. Most drivers will embed the
pads array in a driver-specific structure, avoiding dynamic allocation.

Drivers must set the direction of every pad in the pads array before calling
media_entity_pads_init(). The function will initialize the other pads fields.

pointer to media_entity of the source pad. If NULL, it will use
all entities that matches the sink_function.

constu16source_pad

number of the source pad in the pads array

constu32sink_function

Function of the sink entities. Used only if sink is NULL.

structmedia_entity*sink

pointer to media_entity of the sink pad. If NULL, it will use
all entities that matches the sink_function.

constu16sink_pad

number of the sink pad in the pads array.

u32flags

Link flags, as defined in include/uapi/linux/media.h.

constboolallow_both_undefined

if true, then both source and sink can be NULL.
In such case, it will create a crossbar between all entities that
matches source_function to all entities that matches sink_function.
If false, it will return 0 and won’t create any link if both source
and sink are NULL.

Description

Valid values for flags:

A MEDIA_LNK_FL_ENABLED flag indicates that the link is enabled and can be

used to transfer media data. If multiple links are created and this
flag is passed as an argument, only the first created link will have
this flag.

A MEDIA_LNK_FL_IMMUTABLE flag indicates that the link enabled state can’t

be modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then
MEDIA_LNK_FL_ENABLED must also be set since an immutable link is
always enabled.

It is common for some devices to have multiple source and/or sink entities
of the same type that should be linked. While media_create_pad_link()
creates link by link, this function is meant to allow 1:n, n:1 and even
cross-bar (n:n) links.

The only configurable property is the MEDIA_LNK_FL_ENABLED link flag
flag to enable/disable a link. Links marked with the
MEDIA_LNK_FL_IMMUTABLE link flag can not be enabled or disabled.

When a link is enabled or disabled, the media framework calls the
link_setup operation for the two entities at the source and sink of the
link, in that order. If the second link_setup call fails, another
link_setup call is made on the first entity to restore the original link
flags.

Media device drivers can be notified of link setup operations by setting the
media_device.link_notify pointer to a callback function. If provided, the
notification callback will be called before enabling and after disabling
links.

Entity drivers must implement the link_setup operation if any of their links
is non-immutable. The operation must either configure the hardware or store
the configuration information to be applied later.

Link configuration must not have any side effect on other links. If an
enabled link at a sink pad prevents another link at the same pad from
being enabled, the link_setup operation must return -EBUSY and can’t
implicitly disable the first enabled link.

Before using this function, media_graph_walk_init() must be
used to allocate resources used for walking the graph. This
function initializes the graph traversal structure to walk the
entities graph starting at the given entity. The traversal
structure must not be modified by the caller during graph
traversal. After the graph walk, the resources must be released
using media_graph_walk_cleanup().

Mark all entities connected to a given entity through enabled links, either
directly or indirectly, as streaming. The given pipeline object is assigned
to every entity in the pipeline and stored in the media_entity pipe field.

Calls to this function can be nested, in which case the same number of
media_pipeline_stop() calls will be required to stop streaming. The
pipeline pointer must be identical for all nested calls to
media_pipeline_start().

Indicates that the interface is connected to the entity hardware.
That’s the default value for interfaces. An interface may be disabled if
the hardware is busy due to the usage of some other interface that it is
currently controlling the hardware.

A typical example is an hybrid TV device that handle only one type of
stream on a given time. So, when the digital TV is streaming,
the V4L2 interfaces won’t be enabled, as such device is not able to
also stream analog TV or radio.