Hi,

I've spent a few days investigating Steam Link latency on a range of devices with a number of different settings. I'd like to bring some findings into discussion and hopefully gain a better understanding.

Looking at the streaming stats, my host encoding and frame transfer take <2-3ms. (2080Ti NVENC, 1G ethernet)

However, the display latency on my Raspi4 tends to be >15ms with almost any setting, going into >100ms at 4k. The only reliable setting to keep display latency under 16ms is H.264 Fast or Balanced with 5Mbit bandwidth limit.

My only reference to the performance of hw decoders comes from Nvidia's developer reference, which notes that a Pascal-generation GPU will decode FullHD H.264 at 658fps, FullHD HEVC at 720fps. Essentially, a PC streaming client with a relatively new Nvidia GPU should by those specs have a display latency of <2ms at FullHD. I'm not expecting this level of performance from a Raspi4, but merely noting the data point.

The specs of Raspi4 list capable of 4k60fps HEVC decoding and 1080p60fps H.264 decoding. By that definition, a full frame cannot take more than 16ms to decode.

So here are the questions:
1) Can FullHD HEVC decoding be expected to happen in 1/4 of 4K frame time? If yes, then the decoding part of display latency for FullHD HEVC should be around 4ms.

2) Do the HW decoders in Raspi4 have bandwidth limitations? Display latency does seem to go up with the stream quality.

3) Assuming Steam Link uses vsync, what happens when frames take >16ms to decode and display? Are they queued in a buffer (which should lead to an experience of increased input latency), or is every second frame dropped (leading to a 30fps video stream)?

4) What other factors are involved in display latency besides frame decoding? For example, is a decoded frame copied to a separate buffer for displaying?

5) What happens on a Raspi4 if a 1440p stream is enabled on H.264? (Since the HW decoder only supports up to 1080p.)

A humble thanks for any insights!