I contacted Adobe support and sent them a video performing the tasks that cause this. The rep said they needed to explore it and then closed the case without a response or solution.

I reached out again and they said they need more time to research the issue. They did state the behavior was "expected", but did not elaborate on whether that meant it was a known issue, or if it's intended to be distorted like that. They said they would reach out again.

I'm using Windows 7, PS CC 2017, but I've been able to reproduce this on every version of PS you can install with CC. If I rotate a live shape, this issue doesn't happen, but it also anti-aliases and I don't know how to turn that off. If I create a square and rotate with either free transform or image>rotate, it distorts. If I convert it to a smart object first, it still distorts.

Increasing or decreasing the resolution makes no difference. Increasing or decreasing the dimensions of the square actually DOES seem to make a difference. I found a 1000px by 1000px square creates distortion on the vertical edges, but a 1024px by 1024px square creates distortion on the horizontal edges.

The circumstance where this issue would have a significant impact on anyone's work flow or final result are probably fairly limited, but for those other cases, it's quite perplexing/annoying. If you can reproduce this, please share and report to help find a resolution.

They have closed my case again without a resolution. Put on your tin foil hats. I reached out again. I don't know if there's a language barrier or just a serious fault in how Adobe approaches problems, but it leaves room for a lot of doubt.

I'm getting emails that say my case is closed because the issue has been resolved. I contacted support a third time. They said quote, "this bug has already been fixed if you wan we can keep the case in process,but as such we have not received any reply from engineering,hopefully this will be fixed in the upcoming versoni"

I'm saying I don't understand how the issue can be resolved if I have the latest version of PS and I'm still having this issue. They say, "i can understand your concern very well sir ,the case has'nt been resolved the bug has been reported so in the upcoming version this problem can be resolved"

I'm not saying there is a language, or cultural, barrier that greatly diminished my quality of service, but I'm still only guessing on what all this is suppose to me. It seems like, they're aware of the problem, and MAYBE it will get fixed. Well, that's just fantastic. MAYBE I'll keep paying for my subscription. I'm sure Adobe isn't worried about Idle threats anyway. It's not like Affinity is releasing their inexpensive, award winning programs on Windows...

result: two opposite edges of the square will be toothed while the other two edges will be smooth (with perfect 45 degree tilt)

*) I’m now
sure how the image dimensions affect the rotation. But I have the feeling
that in order to test this issue properly the image width and height must be
odd. Then the image will be symmetric around the center.