After the investigation from support it turns out that controller is not supported at all in skyrim.
Crossover supports dinput for controller interaction, while Skyrim does not and is probably using xinput.