Positive self-evaluation is a major psychological resource modulating stress coping behavior. Sex differences have been reported in self-esteem as well as stress reactions, but so far their interactions have not been investigated. Therefore, we investigated sex-specific associations of self-esteem and stress reaction on behavioral, hormonal and neural levels. We applied a commonly used fMRI-stress task in 80 healthy participants. Men compared to women showed higher activation during stress in hippocampus, precuneus, superior temporal gyrus (STG) and insula. Furthermore, men outperformed women in the stress task and had higher cortisol and testosterone levels than women after stress. Self-esteem had an impact on precuneus, insula and STG activation during stress across the whole group. During stress, men recruit regions associated with emotion and stress regulation, self-referential processing and cognitive control more strongly than women. Self-esteem affects stress processing, however in a sex-independent fashion: participants with lower self-esteem show higher activation of regions involved in emotion and stress regulation, self-referential processing and cognitive control. Taken together, our data suggest that men are more engaged during the applied stress task. Across women and men, lower self-esteem increases the effort in emotion and stress processing and cognitive control, possibly leading to self-related thoughts in stressful situations.