The Temple of the Golden Pavilion on a rainy day in May.
The rain didn't keep the crowds away, it just made it worse cause you had to deal with avoiding other peoples umbrellas. The only times Kinkaku-ji has been quiet for me is on cold winters days as long as it's not coated in snow for that will bring the crowds too.