Most egregious to me is that they completely and totally abandoned any and all tvOS design conventions and standard functions.

The video player is now custom.

I cannot tap the remote and see the timeline. I have to swipe up to see the timeline.

I cannot swipe left/right to rewind/ff. I have to swipe up to see the timeline, then I have to swipe up again to select the timeline. Then I can finally swipe left/right to rewind/ff.

I cannot swipe down to see info or change audio/cc settings.

As noted by OP, it is a pain to scroll all the way back to the left to get to the sidebar items if I happen to be over to the right.

Cursor interactions are just plain fucked. You can't swipe to send the cursor across the row now, or up and down. It kind of sticks as you have to keep slowly moving your finger across the track pad and lift when you get where you want to go.

Basically, every fundamental UI interaction has been changed away from the tvOS standard to a worse implementation, leaving YouTube working in a completely different and altogether worse way than pretty much every other app on the tvOS platform.