I've seen Thor once, I've seen The Avengers twice, I have an above-average long-term memory, and I sporadically discuss these movies on forums, so I probably remember more plot details than the median viewer.

If the movie doesn't work for people who have not seen an MCU movie before never mind all the relevant MCU movies multiple times, then the movie doesn't work in my opinion. There are plenty of sequels out there that can be appreciated by someone who didn't see the original.

Above average long term memory? You couldn't remember Selvig from the Avengers movie and he was a huge part of it.

The Dark Knight trilogy works better if you watch them all. There are things that happen (Rachel breaking up with Bruce and her death) that deeply affect the following movie.