"The smart money is you have to look at everybody's age. I'll put myself at the top of the list. Sooner or later, they've got to start over and get somebody young," he says, adding, "I'm not there with them yet."

And while Downey does have peers amongst his "Avengers" co-stars (Mark Ruffalo is 46 and Jeremy Renner is 43), superhero movies tend to skew in the other direction, age-wise.

"When you really think about it at the end of the day, these films are entertainment largely driven by young folks. There is something very honorable about that," he says.

But Downey won't be putting the suit into storage just yet, and the star is ready to start filming "Avengers: Age of Ultron."

"This one is a very ambitious sequel. If you read it, it's dense, it's smart. Joss [Whedon] is a really smart guy," Downey says of the director. "My 2-year-old is crazy about Hawkeye, and I think Jeremy [Renner] has a lot to do with the plot. There's always so many plates to spin to get these things half-right, and I'm really excited about this one."