I'm posting this as already solved so it can be useful for others.
If you stream Skyrim (in my case the Special Edition) and there is no mouse attached to the server machine the game gets confused and you end up with a stuck scroll wheel that will always scroll down making the game unplayable.
The solution is simply to attach a mouse to the host machine (eventually restart the game).

I don't know if this can be fixed in the Steam Link firmware (or if it's something pertinent only to Bethesda) but it would be nice to be able to run a truly headless system.