It’s not just the victims of bullying that experience long-term consequences; bullies themselves are also at risk of mental health issues later in life.

In a study published in JAMA Psychiatry, researchers report that bullying can have serious consequences on childhood development, and shouldn’t be dismissed as simply a playground rite-of-passage.

Starting in 1993, the scientists followed over 1,400 children at three different ages — 9, 11 and 13, and interviewed them and their caregivers every year until the kids turned 16.

Based on the interviews, they categorized the kids into four groups: victims only, bullies only, both bullies and victims, or neither. To determine the long-term effects of bullying, the researchers re-interviewed the participants when they were ages 19, 21, 24 and 26, and evaluted them for a wide range of different psychiatric disorders.

“Bullying is not just a part of childhood, or some sort of a harmless activity between peers. This is actually something that has very detrimental, and very long lasting effects,” says study author William Copeland of Duke University Medical Center in Durham, North Carolina.

All three groups who reported being involved in bullying experienced some long-term psychiatric effects in the form of anxiety, depressive, or antisocial personality disorders, or some type of alcohol or marijuana abuse. After controlling for family hardships that might also make these mental health issues more likely, the researchers found distinct patterns of psychiatric problems that distinguished the bullies from their victims. Victims of bullying were nearly three times as likely to have issues with generalized anxiety as those who were not bullied, and 4.6 times as likely to suffer from panic attacks, or agoraphobia, in which they felt trapped or had no escape, compared to those who were spared bullying. Bullies themselves showed a four times higher risk of antisocial personality disorder as adults compared to those who did not bully others, and children who reported being both bullies and victims seemed to fare the worst of all; these participants showed a nearly five times greater risk of depression as young adults compared to those who had not both given and received bullying behavior, and a 14.5 times greater risk of having a panic disorder. These effects also showed some gender differences; women had a dramatically higher risk, at nearly 27 times, of having agoraphobia, while men showed an 18.5 times greater prevalence of suicidal tendencies.

“For bullies, it’s a completely different kind of problem,” says Copeland. “With the victims, it is all related to their emotional functioning. For the bullies, they had higher rates of antisocial personality disorder, which is kind of related to criminal behavior, so they’re having completely different problems in adulthood than the victims.”

The findings highlight the importance of considering school- and peer-based factors when kids are struggling, says Copeland. Many counselors, and research studies on the subject, focus on the home situation when children start acting out or becoming more withdrawn. Children’s relationships with their parents or maltreatment in the home are among the more obvious factors that affect behavior, but peer relationships may be just as critical in assessing young children’s development.

“What this study really suggests is that what goes on at school, and what goes on between peers, may be just as important in understanding their long-term function as what goes on at home. In childhood, when kids are in school, they spend a lot more time with their peers than they do with their parents so we should not be so surprised about this,” says Copeland. “When we see kids having trouble, we tend to ask them about things going on at home and we don’t tend to ask them how they’re getting along with their peers and whether they’re the victim of bullying. I think we need to rethink that a bit.”

Taking bullying more seriously, for example, as the potential seed for mental health and behavior problems in adulthood, could lead to better interventions and lower long-term health costs. Childhood experiences lay an important foundation for the type of people we become, and how youngsters interact with their peers is an important part of that dynamic.