router works as the access point, all devices connect to it, not directly to each other

if the host has wifi, you could enable internet connection sharing, and have the link connect to the host directly like that

Wi-Fi Direct IS the devices connecting directly to each other. It's an Wi-Fi standard.
If I did it though your method, I fear it would all go through the router anyways, causing even more delay.
I hope an employee who actually works on Steam Link tells me if I'm wrong about this.