Gandalfs staff is broken during a battle or some event in these movies and Radagast gives Gandalf his staff at the end and "resigns" from his wizardly duties in favor of simply existing with his animals.

It would explain why he's not around in LOTR and it would be a good emotional scene on different levels, showing how the white councils "list of allies" is already growing thin.