We need a mechanism to transport extra data in a GstBuffer. Unlike caps the content is likely to change for each buffer.

Examples:

Pan and scan (video subregion) position information

interlacing markers (top field, bottom field)

theora granulepos, instead of slapping it in the 'offset' field

MPEG Decoding timestamps, which we currently never send along.

Feature-Extraction elements can add feature data

Some image capture sensors can provide histograms with the frames

Some formats (mpeg-ts from HDV cameras) provide date/gain/shutter-speed/scene-change at a regular interval but not for every frame (around 10 times per second).

packet priority and timeout could be used by network plugins (such as DCCP) to send more important packets when congestion is detected.
Ideas:

GstStructure in GstBuffer

GstTaglist in GstBuffer

syncronized downstream events

Store GstStructures in GstIndex
GstStructure is also use in GstTaglist. The structure can be hierarchical. We could require that the producer uses its element name as a root tag. This way the taglist could carry multiple data-sets identified in a unique way. Elements that can use the data need to know the name of the element that created it.