G, who was refused entry to St Gregory's Catholic Science College in Kenton, Harrow, north London, in 2009 because he wears his hair in cornrows, outside the high court. Photograph: Lewis Whyld/PA

A school's anti-gang ban on unconventional hairstyles has resulted in "unlawful, indirect racial discrimination which is not justified", the high court has ruled.

The test case decision is a victory for the family of African-Caribbean teenager "G", who wears his hair in cornrow braids as part of a family tradition.

G, who cannot be named, and his mother challenged a refusal by St Gregory's Catholic Science College in Kenton, Harrow, north London, to let G through the school gates with his braids in September 2009, when he was 11.

Mr Justice Collins, sitting in London, ruled that the hair policy was not unlawful in itself, "but if it is applied without any possibility of exception, such as G, then it is unlawful".

He said in future the school authorities must consider allowing other boys to wear cornrows if it is "a genuine family tradition based on cultural and social reasons".

Even though the family's application for judicial review was successful, G, now 13, does not wish to return to the school, which he left in tears on his first day.

"This is an important decision," said G's solicitor, Angela Jackman, after the hearing. "It makes clear that non-religious cultural and family practices associated with a particular race fall within the protection of equalities legislation."

The judge emphasised that the school's "short back and sides" hair policy was perfectly permissible and lawful, but exceptions had to be made on ethnic and cultural grounds.

He stressed that the school was "not in any way racist" but had made "an honest mistake" in failing to allow for exceptions, adding: "The school has had glowing Ofsted reports and there is no question that it is an excellent school."

The judge said headteacher Andrew Prindiville had justified the policy as necessary to stop the gang culture prevalent in the area, in which haircuts were used as badges of membership, coming into the school.

Cornrows were not necessarily gang-linked but other styles, like the skinhead haircut, might well have that connection, the judge said.

The fear was that allowing exceptions to the "short back and sides" rule would undermine the anti-gang policy.

But the judge pointed out that exceptions were already made for Rastafarians and Sikh boys who wore hair beyond the collar, and similar exceptions should be made for African Caribbeans.

The judge said G's family was not alone in regarding cornrows as part of their culture: "There are, on the evidence, other African Caribbeans who take the same view."

The judge refused the head teacher and governors permission to appeal to the court of appeal, but they can still go to the appeal judge directly to ask them to consider the case.

The judge stressed that he was not ruling on whether the exclusion of G in 2009 was unlawful. It had been suggested that G's family might bring a county court damages action over the case.

That would be the time to decide whether or not the school had dealt with G's desire to wear cornrows in an unlawful manner, said the judge.

Jackman said the judge had found the school's policy unlawful as it applied to African Caribbean boys with G's beliefs because it indirectly discriminated on race grounds.

She added: "For G, wearing his hair in cornrows is a fundamental cultural practice which would have had no adverse impact upon the school. His wishes, however, were dismissed by the school without any consideration. Whilst schools face the challenges of maintaining good discipline, a community environment and their particular ethos, this case is a reminder that they must do so within the boundaries of the law."