Because love is not consent.
Lets pretend Hanako DIDN'T want to have sex with you, she still loved you and wanted you to be happy, even at her own expense.

But yeah, overall I do agree with you, it wasn't rape and this is made painfully obvious by the end of the story. As for what we think while it's happening? well, that's just one reason the games special, everyone interprets these things differently and the writing was good enough to hint at things and let your mind go into overdrive until things were explained.