We can probably all recognise those attention-seeking people in our lives – and increasingly it seems in politics and pop culture – who have a hugely inflated sense of their own importance and abilities, combined with a relative disregard for other people’s. Psychologists call them narcissists, after the character Narcissus from Greek mythology, who fell in love with his own reflection.

When you meet someone like this, their bravado can be alluring at first, but soon the sheen wears off as their look-at-me antics and disdain for others becomes increasingly apparent. You’ve probably come to find the narcissist in your office or family (or on your TV screen) arrogant and annoying. If so, that’s understandable, but actually some of the latest research findings in this area suggest that the most appropriate response to narcissists is probably pity, and maybe even kindness.

Narcissists are quick to associate self-related words like “me”, “mine” or “myself” with unpleasant words like “pain”, “agony” and “death”

Consider first the consistent discovery that beneath their hubris and egomania, many narcissists actually suffer from chronic low self-esteem. This has been demonstrated in many different ways, including using a version of the “implicit association test”, which in this context measures how readily people associate words referring to the self with pleasant or unpleasant words. One telling study found that highly narcissistic people said they had high self-esteem, yet when tested in the lab, they were very quick to associate self-related words like “me”, “mine” or “myself” with unpleasant words like “pain”, “agony” and “death”.

Another imaginative method that’s uncovered the inner fragility of the narcissist is the so-called bogus pipeline technique. Some of the participants are wired up to physiological recording equipment that they’re told will reveal whether they are lying, while others in a comparison control condition are connected to the equipment but they’re told it has been turned off. A study involving 71 women found that the narcissists among the participants reported having much lower self-esteem when they felt their lies would be unmasked, compared with those narcissists in the control condition. Indeed, they even reported lower self-worth than women who weren’t narcissists.

Increasingly, this picture of the narcissist as over-compensating for their private self-doubt is being supported by findings from brain imaging research. For example, one study involved male teenagers having their brains scanned while they played a collaborative computer game called cyber-ball. When their team-mates ignored them, the more narcissistic participants didn’t say it bothered them any more than the others, and yet their brains showed unusually high activity levels in regions that have previously been associated with the experience of social and emotional pain.

Narcissists have a deficit in their reward circuits that may explain why they are always trying so hard to shore up their self-confidence

More recently neuroscientists at the University of Kentucky used a different kind of scanning technology to investigate the density of connective tracts in different parts of the brains of their participants. The research published earlier this year showed that the higher participants scored on a questionnaire measure of narcissism, the less connective tissue they had between the medial prefrontal cortex – a brain region associated with thinking about the self – and the ventral striatum, which is a region tied to the experience of reward and pleasure. The researchers said this “internal deficit in self-reward connectivity” might make it difficult for narcissists to think positively about themselves and that it could explain why they are always trying so hard to get attention and shore up their self-confidence.

But the fact that narcissists are inwardly fragile is not the only reason to feel sorry for them. Another line of research suggests that behaving the way they do is going to make life stressful for them. A Swiss study assessed hundreds of people several times over several six-month periods, including measuring their narcissism and their experience of stressful events. The results showed that higher scorers in narcissism tended to go on to experience more stress in life, such as illness, accidents and relationship breakups. Based on this, the researchers at the University of Bern concluded that “narcissism is maladaptive for the individual, because narcissistic individuals generate adverse events in their lives.”

This is particularly bad news, considering that they may be more sensitive than most to the adverse effects of stress. For example, a team led by Joey Cheng at the University of Illinois at Urbana-Champaign asked 77 female undergrad students to keep a diary of their negative emotions and took samples of their saliva to look for signs of a biological reaction to stress (specifically, cortisol and the protein alpha-amylase). They found that those who scored high in narcissism showed increased physiological signs of stress the more negative emotions that they experienced, whereas non-narcissists did not – findings that suggest that narcissists are surprisingly thin-skinned and sensitive.

At the same time as recognising their fragility, it’s worth remembering that narcissistic people do have some redeeming qualities; in some contexts they can be unusually persistent in the face of set-backs, no doubt to prove their worth to themselves and others, and there’s evidence that creative teams can benefit from having a narcissist or two to help inspire a little healthy internal competition. There’s even some evidence that with a little encouragement – asking them to take other people’s perspective – narcissists can be nudged into showing greater empathy.

Taking all these findings together there’s good reason to try, if you can, to be a little more patient – more loving even – with the narcissists in your life. More likely than not they are overcompensating for their deep-rooted self-doubts, and while they might seem smug and to be stealing all the lime-light, the chances are that longer-term they will not fare well, especially if events turn against them.

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.