The standard assumption is that women's lives have dramatically improved over the last 50 years. They have considerably more personal freedom; and opportunities for education and employment have been transformed. As a result they have much greater financial independence, which has given them more power to shape their lives. So far, so easy.

But something odd is going on that no one can explain. These huge social changes are not making women happier, and, according to several significant studies, women's happiness relative to men's has declined in the last 25 years. This includes women of all age groups, and it is evident in many countries, particularly in the US and the UK.

Let's start with the most alarming evidence. It comes from the West and Sweeting study of 15-year-olds conducted in exactly the same place in Scotland in 1987, 1999 and 2006. When the 1999 results were published, there was concern that the incidence of common mental disorders such as anxiety, depression, panic attacks and anhedonia (loss of capacity to experience pleasure) had significantly increased for girls from 19% to 32%. The increase for boys was much smaller, at only 2%. But the latest set of results are even more dramatic. There has been an increase for both sexes: boys are now on 21%, and girls are at a staggering rate of 44%.

The rate of increase is appalling. Over a third of girls agreed "they felt constantly under strain"; those who "felt they could not overcome their difficulties" had more than doubled to 26%. The number who agreed with "thinking of yourself as a worthless person" had trebled between 1987 and 2006. These findings could partly explain the recent reports of sharp rises in girls' binge drinking and aggressive behaviour.

The first thought is that perhaps this gender gap is a teenage thing. Other studies showing a marked increase in mental ill-health of teenagers have prompted speculation that the transition to adulthood now is much more difficult and demanding. But the gap in mental ill-health between men and women is just as striking in other age groups; an NHS study published this year showed that between 1993 and 2007 common mental disorders had risen by a fifth for women aged between 45 and 64 (there had been no change in men), and among the over-75s, they were twice as likely in women as men.

Various explanations are put forward. Women's levels of serotonin are more vulnerable, it has been suggested, but that doesn't explain the change over time. Women are struggling with work and family, looking after their elderly parents, or coping with empty nest after children have left. Two American academics checked all the data from the US and the European Union to try to hunt down the explanation.

Stevenson and Wolfers found that American women – of all social classes, ages and whether they worked, stayed home, had kids or did not – had seen a decline in happiness since the early 70s. Thirty years ago, women reported higher rates of subjective wellbeing than men in the US. This advantage has been entirely eroded, and in many instances it is now men who are happier than women. So how did women manage to end up, after a generation of advances in gender equality, less happy typically than their mothers at their age?

There are no easy answers, conclude Stevenson and Wolfers. They pose the extraordinary question: "Did men garner a disproportionate share of the benefits of the women's movement?" They suggest "perhaps the wellbeing data point to differential impacts of social changes on men and women, with women being particularly hurt by declines in family life, rises in inequality or reductions in social cohesion". One finding they highlight is that women's satisfaction with their financial situation has declined while men's has remained stable – one possibility is that there has been a change "in the reference group" or expectations for women so that their lives are more likely to come up short.

This latter is key to the work of another American psychologist, Jean Twenge, whose most recent work has been to analyse what she describes as a "narcissism epidemic" in the US that is disproportionately affecting women. Her meta-analysis covered 37,000 college students. It found that in 1982, 15% got high scores on a narcissism personality index; by 2006 it was 25% – and the largest share of this increase was women.

The narcissist has huge expectations of themselves and their lives. Typically, they make predictions about what they can achieve that are unrealistic, for example in terms of academic grades and employment. They seek fame and status, and the achievement of the latter leads to materialism – money enables the brand labels and lavish lifestyle that are status symbols. It is the Paris Hilton syndrome across millions of lives.

Twenge points to the fact that in the 1950s only 12% of college students agreed that "I am an important person", but by the late 80s it was 80%. In 1967, only 45% agreed that "being well-off is an important life goal", but by 2004 the figure was 74%.

The problem, Twenge believes, derives in part from a generation of indulgent parents who have told their children how special they are. An individualistic culture has, in turn, reinforced a preoccupation with the self and its promotion. The narcissist is often rewarded – they tend to be outgoing, good at selling themselves, and very competitive: they are the types who will end up as Sir Alan's apprentice. But their success is shortlived; the downside is that they have a tendency to risky behaviour, addictive disorders, have difficulties sustaining intimate relationships, and are more prone to aggressive behaviour when rejected.

The narcissism of young women could just be a phase they will grow out of, admits Twenge, but she is concerned that the evidence of narcissism is present throughout highly consumerist, individualistic societies – and women suffer disproportionately from the depression and anxiety linked to it.

This is what alarms psychologist Oliver James. He is working on an updated version of his pioneering Britain on the Couch, which first argued that mental ill-health had increased despite more wealth. He worries that the Scottish teenage girls are the "canaries" down the mines, giving powerful indications of a set of social influences that are deeply damaging their wellbeing. He points to the pressures of a "consumerised, commercially driven version of femininity" that puts huge emphasis on girls' appearance.

Girls are more compliant and eager to please – that is how they have always been socialised – but now the dominant social expectations of them are deeply destructive of their happiness. Breast augmentation quintupled in 2006 in the US, Twenge points out. The expectations of girls and women have multiplied and intensified – on every front, from passing exams to looking good and having more friends and better photos on Facebook. Technology proliferates the places in which one is required to self-promote.

One possibility is that women's identity has always been framed around relationships – as mothers, daughters, wives, friends and sisters. "Relationality" is still central to how women see their lives, and yet it is entirely at odds with an individualistic, intensely competitive, narcissistic culture. Women, brought up to seek social approval, battle between competing frames of reference, and many end up feeling failure and inadequacy on multiple fronts.

Sign up for the Guardian Today

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.