No, Quicksilver/Scarlet Witch have never been X-Men and they are completely Avengers characters (over 90% of the comics they've been in are Avengers related, especially Wanda). Fox can use them solely because they're mutants, but in terms of their actual role in the comics, they have very little connection to the X-Men side of stories other than a few story arcs every other decade.

Quicksilver was a part of X-Force in the 90's, but that's the largest connection they've ever had to them up until the past decade.

He seems to be Magneto's ally in DOFP, which fits his role in the X-Men comics.