Gandalf and Saruman, you know where they were. The blue wizards Tolkien never really went into. Gandalf visited Radagast's home near the beginning of the ring trilogy and it was completely deserted, whereabouts unknown.

Look at it this way, he wrote The Hobbit to read to his children and it took family and friends a long time to get him to publish it and even longer to write the trilogy, thank God for his son Christopher publishing so much else postmortem.