DM365 LeopardBoard network video streaming latency test

A simple test to get a feel for the amount of system latency when RTP network streaming live video from a LeopardBoard 365 to a PC. A website that displays a running stop watch (http://stopwatch.onlineclock.net) is captured by the target hardware camera, streamed over the network using RTP and displayed on the PC. The windows on the PC are arranged in such a manner that both the stop watch web page and the live video screen are visible. The PrintScreen feature is used to grab a snapshot of both screens at the same time. The difference between the time on the stopwatch and the time displayed on the live video screen shows the approximate system latency.

PC receiving video only pipeline

Before starting PC video playback, you may have to adjust the caps parameters to udpsrc in the gst-launch command below. Look around 40 lines down from the output of the target hardware gst-launch video transmit command for a line similar to:

RidgeRun PrintScreen based latency results

Some of the samples were capture part way through a screen refresh, making the values hard to read, so those samples were discarded.

Example PrintScreen (the video capture driver was configured for LI-LBCM5M1 and I was using a LI-5M03 so it produced a mirror image).

Captured latency:

Stopwatch(sec)

Live Video(sec)

Latency(ms)

Frame Latency(frames, assuming 30 fps)

58.31

58.22

90

3

34.68

34.32

360

11

51.91

51.85

60

2

04.91

04.74

170

6

23.62

23.45

170

6

39.39

39.13

260

8

01.22

01.14

80

3

Latency measurement errors

Using the on screen timer/clock approach can lead to measurement inaccuracies, including, but not limited to:

monitor vertical refresh cycle

accuracy in the timer/clock being updated

delays in the updated timer/clock being display on the monitor

Post power-on commands

ifconfig eth0 # learn address to use to telnet to target hardware
inetd

Telnet to the target hardware and run top if you are interested in monitoring the ARM CPU load.

Custom hardware latency measurement ideas

GPIO connection

A new gpio-assert GStreamer element could created that after a certain number of buffers flow (to get past the pipeline startup delays), the element could put a special color in for the first 8 pixels and assert a GPIO signal. The gpio-assert element goes right after the v4l2src element.

On the receiving side, right before the video sink element, a new gpio-monitor element could be added. It spawns a thread to watch for a GPIO to change. When it changes, the element records the GStreamer clock and starts monitoring the video frames for the special color (which may not be an exact match since the frame has been compressed and decompressed). When the frame is detected, gpio-monitor again reads the GStreamer clock, subtracts the two values, and reports the latency.

The trick is to find an available GPIO on the receiving device that supports interrupts.

Optimization ideas

rtph264depay has a queue setting as well as rtspsrc. Setting rtspsrc latency to zero the video frames are not very smooth.
Increasing the value to around 30 causes the video is smooth and appears low-latency (not yet tested with camera / screenshot). udpsrc likely has a similar parameter. Also the introducing queue elements adds some latency (hopefully configurable).