I think the movie is getting crowded by just introducing Scarlet Witch and Quicksilver so I wouldn't want any other heroes involved. They won't be used well and just take away time for the Avengers to shine, and people wanted more from some of them already in the first movie.