(CNN) -- Using marijuana at an early age could have long-term consequences on your brain and it may even lower your IQ, according to a new study in the Proceedings of the National Academy of Sciences.

Researchers found that compared to nonusers, people who smoked marijuana starting as early as age 14 have less brain volume, or gray matter, in the orbitofrontal cortex. That's the area in the front of your brain that helps you make decisions.

"The younger the individual started using, the more pronounced the changes," said Dr. Francesca Filbey, the study's principal investigator and associate professor at the School of Behavioral and Brain Sciences at the University of Texas at Dallas. "Adolescence is when the brain starts maturing and making itself more adult-like, so any exposure to toxic substances can set the course for how your brain ends up."

Researchers also found increased brain connectivity in chronic users. Connectivity, when different parts of the brain connect to each other, is important for adaptive learning abilities. It also helps your mind make associations. This wiring of the brain starts to deteriorate with chronic marijuana use.

"Too much or too little of anything isn't good. There needs to be an equal balance," said Filbey.

There were 48 marijuana users enrolled in this study. All started smoking between age 14 and 30. The average age of the person enrolled in the study was 18. On average they used marijuana three times a day. Most said they had been using it for 10 years, although some had been smoking pot for three decades.

Scientists compared this group to a group of 62 nonusers of the same basic age and gender. All gave urine samples. All had an MRI scan and all went through IQ testing.

Filbey said the people who regularly used marijuana had IQ's that were five points lower, on average, than the nonusers in the study, although there is no definitive proof that marijuana alone was to blame for the lower IQ.

"While our study does not conclusively address whether any or all of the brain changes are a direct consequence of marijuana use, these effects do suggest that these changes are related to age of onset and duration of use," Filbey said.

Dr. Susan Weiss, associate director for Scientific Affairs at the National Institute on Drug Abuse said the study provides more strong evidence about the dangers of marijuana.

"This is a complex and interesting study that adds to the growing body of evidence that heavy marijuana use, particularly at a young age, is linked to significant adverse brain changes," said Weiss. "This study showed that the orbitofrontal cortex, an area involved in reward, decision making and motivation, was smaller in heavy users and that other brain circuits were enhanced, likely to compensate for the diminished function in that region. Further prospective studies are needed to clarify this, but these mounting scientific findings certainly challenge the widespread belief that marijuana is a harmless drug."

The National Institute on Drug Abuse helped fund the study.

Mason Tvert, director of communications for the Marijuana Policy Project, said marijuana use does not cause IQ loss.

"Once again, researchers have failed to find any conclusive evidence that marijuana use causes mental health problems. The researchers note their findings are nonconclusive, that they might be skewed by other factors, and that effects, if any, could be temporary, Tvert said. "The study doesn't justify keeping marijuana illegal, nor does it say anything about making it legal. There remains no doubt that marijuana is far less harmful than alcohol to the brain and to the rest of the body. The possibility that marijuana might have some harm for some people -- but might not -- is not a good reason to keep arresting and punishing hundreds of thousands of adults simply for using it."

Filbey next plans to do long-term observational studies with people who do not use marijuana to see if there are underlying factors like pre-existing conditions before exposure to THC, the psychoactive ingredient in cannabis, that could account for some of these effects.