Previous research suggests an advantage for women over men in recognizing faces. (Lewin & Herlitz, 2002; Bowles et al. 2009). However, this ignores the variety of face types and varied experience with these faces. McGugin et al. (2012) showed how using performance for several object categories led to a different interpretation of sex differences, one emphasizing the role of experience, compared to an approach where differences are obtained for a single category, and interpreted as due to general visual or cognitive mechanisms (Dennett et al., 2010). Here, we apply similar logic to faces to explore sex differences in face recognition that may be influenced by relative experience with face categories. We developed the Vanderbilt Face Expertise Test (VFET), based on the format of the Vanderbilt Expertise Test (VET; McGugin et al. 2012) and the Cambridge Face Memory Test (CFMT; Duchaine & Nakayama, 2006). In 480 participants (181 male), we measured recognition performance for 4 face categories (Barbies dolls, Transformer action figures, Caucasian males, Caucasian females) and cars. For each category, participants studied 6 targets and found them among pairs of distractors on 48 trials. Women outperformed men with Barbie faces (Sex * Category interaction, F(1,355)= 14.92, p=.0002) and men outperformed women with Transformer faces (F(1,353)= 5.92, p=.016). Moreover, multiple regressions suggested qualitative differences in performance on toy categories; men's performance on toy faces was predicted by performance on other toy categories, whereas women's performance was predicted by performance with human faces. While prior work with faces finds an advantage for women or no gender effect, we find a face category for which men outperform women, suggesting that experience may drive prior gender effects. Furthermore, the results suggest that experience with faces may not only influence overall performance, but may result in qualitative differences in how men and women recognize faces.