Unable to progress. Can't continue the story

I am a level 50 Swashbuckler, and finished up Haunted Skyway. After defeating The Deacon, I had gone back to Avery's office and listened as he told me to speak to Zu Rho, in Zu Rho's teahouse, in Hamamitsu garden.

Well, entering the teahouse, I noticed Zu Rho standing there (the brown colored Samoorai) but no exclamation mark. Or question mark. There is nothing that I can do, even in the vicinity of Zu Rho. My quest marker points me to some direction near the edge of the island, but after reading posts about wonky quest markers, I thought nothing of it.

But I am stuck. There is nothing I can do. I still do not have access to Khotan or Subata skyway, so there is nothing I can do. I can't finish promoting El Toro.