Rewinding

(This page talks only about sink rewinding, but supposedly source rewinding works mostly in the same way. I'm not very familiar with that.)

Rewinding is mostly used for making changes in the audio hearable as quickly as possible. For example, if volume changes, the change should be hearable immediately. The amount of buffered audio may be large, for example 2 seconds, so without rewinding the changed volume would be hearable only after the 2 seconds of buffered audio have been played. With rewinding, the data in the buffer is rewritten using the new volume.

Buffers

A useful way to think about the system is that there are three buffers: the sink buffer, the render_memblockq and the sink input implementor's buffer. Each buffer has a read and a write index that normally move forward, but on rewinds one or both of them move backwards. The sink read index never moves backwards. You can think the sink read index as the point where the digital audio is converted into analog signal - there's no way to rewrite that data.

The three buffers can be thought as infinite, so don't think them as ringbuffers (even if the implementation is a ringbuffer). They don't contain an infinite amount of data, though. The word "infinite" means just that the read and write index grow without limit (the indexes are implemented as 64-bit integers, so they are virtually unlimited). The sink buffer has a hard limit on how far ahead the write index can be from the read index. In other words, the sink buffer has a defined size. The sink buffer size is also the maximum rewind amount that the sink supports, and that is stored in pa_sink.thread_info.max_rewind.

The max_rewind variable is important, because the render_memblockq and the sink implementor have to keep at least that much of old data buffered. "Old data" means data that has already been given to the sink. If a rewind happens, the sink may need to ask for that old data from the sink input again.

In addition to the old data, render_memblockq may also have some data buffered that has not yet been given to the sink. This happens only when the sink implementation gives a larger chunk of data than requested, so there's at most one extra chunk buffered in the render_memblockq, so usually it's only a small amount, but in theory that chunk can be arbitrarily large, so don't assume that render_memblockq always contains only a small amount of data (in addition to the old data that has to be kept stored).

The sink input implementor's buffer has to contain enough old data to cover the sink buffer and whatever extra there is in the render_memblockq. On top of that, the sink implementor may or may not have some extra data of its own buffered. If the implementor can generate an arbitrary amount of data at any time (like module-sine), then there's no need to have any extra data buffered. However, usually the data is received from somewhere else, like from clients using the native protocol, and in those cases there needs to be buffering to prevent underruns.

There's actually another buffer between render_memblockq and the sink input implementor (in the example above the native protocol stream): if a resampler is used, it may sometimes have some data in its leftover buffer. BUG: That buffer is not currently taken into account when rewinding (https://bugs.freedesktop.org/show_bug.cgi?id=53911).

Filter Sinks

With filter sinks there may be an arbitrary amount of sinks and sink inputs stacked. The filter sinks may or may not have a buffer of their own, but the sink inputs certainly will each have a render_memblockq. The "real" sink inputs, for example native protocol streams, need to keep enough old data buffered to cover all audio that is buffered in the stack of filter sinks.

BUG: Currently this is broken (https://bugs.freedesktop.org/show_bug.cgi?id=53709). The sink input implementations assume that storing max_rewind plus the extra data in their own sink input's render_memblockq is enough. That's OK: the assumption really should hold. The problem is that it doesn't. The max_rewind of a filter sink is incorrect. It's simply copied from the root sink. That's not a good idea, because it ignores the data that is buffered in the filter sink's sink input's render_memblockq, and if the filter sink does any buffering of its own, then also that data is missing from max_rewind. Having a too small max_rewind value means that rewind requests get truncated too much, causing unnecessary latency.

The max_rewind variable is somewhat static. It can change when the latency configuration changes, but it can't change depending on how much data is currently buffered. max_rewind is supposed to tell an upper limit for how much the sink can have data buffered, not the currently buffered amount. Since render_memblockqs can have an arbitrary amount of data buffered, how can a filter sink provide a meaningful value for max_rewind? I think the answer is that the filter sinks should never return more data from their pop callbacks than requested. That way the render_memblockq of the sink input will never contain any extra data. Also the pa_sink_render_* functions need to have a parameter for specifying a maximum size for the rendered data, because the rendering functions are called by the filter sinks, and if the rendering can result in an arbitrarily large chunk that the filter sink has to buffer, then it's still impossible for the filter sink to provide a meaningful max_rewind value.

When the filter sink max_rewind issue is fixed, the "three buffers" abstraction works with filter sinks also. The filter sink can be thought to have just a single buffer with length of max_rewind. When rewinding the filter sink, the write pointer of the sink buffer will move back, and it doesn't matter to whoever issues the request if behind the scenes there are actually multiple buffers, each of which will be rewound recursively until the request is fulfilled.

Requesting and Processing Rewinds

Rewinds are handled in two phases: first a rewind is requested, and then processed. The requests must only come from code in the IO thread that is executed as part of handling messages. This means that requesting a rewind from the main thread is forbidden, but it's also forbidden while rendering data, for example.

The IO thread of a sink runs in a loop that basically does the following things:

Sleep until something happens. This usually means that the backend either wakes up the sink to fill the backend buffer, or with timer-based scheduling, the timer expires. Another common thing is that the sink receives a message.

Process the event that woke up the sink. In case of a received message, this may generate rewind requests, which are merged into one request.

Process the rewind request if there is one.

Fill the sink backend buffer by requesting audio from sink inputs.

The first two bullet points are implemented by pa_rtpoll_run().

BUG: Rewinds are processed also by filter sinks in their pop callbacks. While this should be pretty harmless (there should never be any rewind requests pending when the callback is called), it's useless code and may cause confusion (https://bugs.freedesktop.org/show_bug.cgi?id=53915).

Rewind requests can be made with pa_sink_request_rewind() or pa_sink_input_request_rewind(). pa_sink_input_request_rewind() always calls pa_sink_request_rewind(). The rewind requests must always reach the sink that implements the IO thread (the "root sink"), because processing of the rewinds is always done from that sink, and it can't process rewinds that have not been requested from it. Requests made on filter sinks are forwarded to the master sink by calling pa_sink_input_request_rewind() on the sink input between the filter sink and the master sink.

If multiple rewind requests are made before the processing phase, they are merged so that the biggest rewind request wins.

Sometimes the rewind amount that is requested from a sink is 0, for example if pa_sink_input_request_rewind() was called and the request can be fully satisfied just by discarding some of the data in the render_memblockq of the sink input. Even though there's no need to rewind anything on the sink, pa_sink_input_request_rewind() will anyway call pa_sink_request_rewind() with amount of 0 bytes, because the sink needs to be notified about all rewind requests so that those requests will get processed.

When the sink notices (in the IO thread loop in the sink implementor code) that a rewind request has been made, it rewinds the backend buffer as much as was requested, or if that's not possible, then as much as possible. Then the sink calls pa_sink_process_rewind() with the real amount of bytes that was rewound. pa_sink_process_rewind() will then call pa_sink_input_process_rewind() for all sink inputs.

So, rewind requests are propagated towards the root sink, and rewind processing is propagated outwards from the root sink.

Rewinding a Sink

Doing a rewind on a sink is conceptually rather simple. The sink has a buffer, and there is a read index and a write index. The sound card reads from the read index, and data from sink inputs is written to the write index. When a sink is rewound, the read index is always left alone (what has been played can't be unplayed), so only the write index is involved in the rewind operation. Rewinding the sink by N bytes simply means moving the write index back by N bytes.

Filter sinks make this a bit more complicated, though. Depending on how many layers of filter sinks there are, there are at least two buffers involved: the sink input buffer (render_memblockq) and the master sink buffer. When the filter sink is requested to rewind by N bytes, the write index of the render_memblockq is moved back by N bytes, and if the render_memblockq doesn't have that much data stored, then also the master sink buffer is rewound by the amount that was not available in the render_memblockq.

When the sink has processed the rewind for its own part, it calls pa_sink_input_process_rewind() on all of its sink inputs, passing the amount of bytes by which the sink was actually rewound. If there was no rewind request made on a sink input, then the rewind processing will only move the read pointer of the render_memblockq (to match the moving of the sink write pointer).

Rewinding a Sink Input

The simple case is when there has been no rewind request on the sink input, and pa_sink_input_process_rewind() is called just to move the read index of render_memblockq to compensate the write index moving in the sink buffer, as explained in the previous section. But if there has been a rewind request on the sink input, then things are more complicated.

When a rewind request is made on a sink input, four parameters are given:

nbytes: The amount to rewind. 0 means "as much as possible".

rewrite: Should the sink input implementor rewrite the rewound amount of data? If this flag is set, the side effect is that the write index gets moved back by the rewound amount. Otherwise the side effect is that the write index is flushed (i.e. moved to point to the read index).

flush: Should the render_memblockq contents be discarded (both old data and any data on top of it)? This parameter doesn't affect the read or write index. If rewrite is false, then a flush is implied.

dont_rewind_render: Should the render_memblockq read index (not) be moved?

The parameters are stored in three variables:

thread_info.rewrite_nbytes: If any of the merged requests have rewrite unset, then this will be -1. Otherwise this will be the biggest of the merged rewind requests.

thread_info.rewrite_flush: If any of the merged requests have flush set and request a rewrite of more than 0 bytes, then this flag will be set. A rewrite request of 0 bytes can only happen when the sink has max_rewind of 0 bytes and the render_memblockq is empty, or when thread_info.playing_for is 0 (so the sink input has been just created or having an underrun). In those cases flushing the render_memblockq is probably a no-op. (But is it actually harmful? Why is the check for 0 done if it doesn't matter in any case? Maybe it's just an optimization, avoiding a redundant pa_memblockq_silence() call? But then again, the pa_memblockq_silence() call is only made when the rewrite amount is more than zero... BUG https://bugs.freedesktop.org/show_bug.cgi?id=53923)

thread_info.dont_rewind_render: If any of the merged requests have dont_rewind_render set, then this flag will be set.

(It's hard for me to be sure that the different kinds of rewinds work correctly when merged... But if the current logic works fine, then maybe it's best to not try fixing what is not broken.)

The rewind amount that pa_sink_input_request_rewind() gives to pa_sink_request_rewind() is what has been stored in thread_info.rewrite_nbytes, if rewrite = true. If rewrite = false, then it's a bit complicated to explain, except that in practice it seems that if rewrite = false, then the original nbytes is always 0, which will make the final nbytes equal max_rewind.

Based on the stored variables, pa_sink_input_process_rewind(nbytes=N) will behave as follows (N is the real amount that the sink rewound):

If dont_rewind_render is not set, the read index of render_memblockq is moved back by N bytes.

If rewrite_nbytes is -1 (i.e. there was a rewind request with rewrite = false), then the write index will be reset to the read index.

If rewrite_nbytes is not -1, then the write index of render_memblock is moved back, if the following conditions are met:

rewrite_nbytes is greater than 0, which implies:

the sink supports rewinding or the render_memblockq contains data

playing_for is greater than 0

the actual amount that the sink rewound was greater than 0 or the render_memblockq contains data

The amount that the write index of render_memblockq gets moved back is the actual amount that the sink rewound plus the length of the render_memblockq, or rewrite_nbytes, whichever is less.

The sink input implementor is notified of the rewind with the process_rewind() callback. The nbytes parameter of the callback is the amount by which the render_memblockq write index was moved (or 0 if there was a rewind request with rewrite = false).

Current Usage (as of 2012-08-11)

pa_sink_request_rewind()

The alsa sink does a full rewind when the latency is made smaller. There needs to be a rewind to ensure that the sink buffer doesn't contain more data than what is allowed by the new latency. This is because rewind requests can't be larger than the configured latency, and if there's more data than that in the buffer, a later rewind request may end up being too small. BUG (minor performance issue): A full rewind is unnecessary (https://bugs.freedesktop.org/show_bug.cgi?id=54007). BUG (not affecting users): The alsa sink shouldn't need to do this at all (https://bugs.freedesktop.org/show_bug.cgi?id=54006).

This is essentially the same case as removing a sink input, from the sink point of view (from a wider point of view there are much more details to get right, and things aren't currently perfect. BUG: https://bugs.freedesktop.org/show_bug.cgi?id=54182).

Rewind is requested to make the moved sink input hearable in the new sink as quickly as possible. nbytes is the current sink latency, so in practice this is a full rewind (maybe it would make sense to be explicit about it by changing nbytes to (size_t) -1).

Sometimes when pa_sink_get_volume() is called, it's desirable to go all the way to the hardware to ask the current volume. When that happens, it may be that the received value is different than the current sink volume, and in that case the sink volume is updated. That may cause a change in the sink soft volume too, and in that case the sink is rewound to make the new soft volume hearable as soon as possible.

The echo cancel sink input has been just attached after creation. A rewind request is made so that the sink input becomes hearable as soon as possible. BUG (not affecting users): This could actually be handled by the core in PA_SINK_MESSAGE_ADD_INPUT (https://bugs.freedesktop.org/show_bug.cgi?id=54243).

The sink input has been having an underrun which has now ended. A rewind is requested so that the data that has been missing can be added to the sink buffer. BUG: Instead of requesting a full rewind, the code should check how long the underrun has been. Doing a full rewind can cause skipping if the underrun was short and the sink buffer still has valid data from before the underrun started (https://bugs.freedesktop.org/show_bug.cgi?id=54251).

The sink input has been having an underrun which has now ended. A rewind is requested so that the data that has been missing can be added to the sink buffer. BUG: Instead of requesting a full rewind, the code should check how long the underrun has been. Doing a full rewind can cause skipping if the underrun was short and the sink buffer still has valid data from before the underrun started (https://bugs.freedesktop.org/show_bug.cgi?id=54251).

The sink input is being corked. A rewind is requested to remove the old data from the sink buffer. The "rewrite" parameter is true, because when uncorking, we want the playback to continue from where it was before corking, so we don't want to lose the old data. The "flush" parameter shouldn't matter, because it's a full rewind anyway, but setting it to true ensures that there will be no old data left in the render_memblockq. The "dont_rewind_render" parameter is false, because otherwise the render_memblockq write index would get moved past the read index. With these parameters the read and write indexes will be the same after the rewind.

The sink input is being uncorked. A rewind is requested so that the sink input becomes hearable as quickly as possible. The "rewrite" parameter is false, because the head of the sink input implementor's buffer is already at the point where the cork happened (see the previous section). The "flush" parameter doesn't matter, because there's no data in the render_memblockq anyway. The "dont_rewind_render" parameter is true, because the read and write indexes of the render_memblockq are the same. Moving the read pointer would cause silence to be generated, because the write pointer is not moving.

The sink updates its sink inputs' soft volumes, so a rewind is requested to make those changes audible as quickly as possible. BUG (not affecting users): This duplicates the code in the PA_SINK_INPUT_MESSAGE_SET_SOFT_VOLUME handler, so it would be better to just send that message to the sink inputs (https://bugs.freedesktop.org/show_bug.cgi?id=54252).