Individuals exhibiting high levels of physical aggression during childhood are more at risk of criminal behaviors during adulthood regardless of the presence of several risk factors. Aggressive children are also more at risk of using psychoactive substances during adolescence. Studies of the relationship between substance use and crime have provided different conclusions, depending on the substance under consideration and depending on the category of crime. Alcohol consumption has been positively associated with crimes, both against people and against property. Cannabis consumption has been associated positively with crime against property and negatively with crime against people. This present study aimed at investigating the hypothesis that physical aggression manifested in childhood is associated with an increase in criminal charges during adulthood and that association is specific to the type of crime and moderated differentially by alcohol and cannabis use. To this end, 887 male participants from lower socioeconomic status were followed from age 6 to age 25. Multinomial logistic regressions indicate that individuals belonging to a moderate or a high trajectory of physical aggression during childhood are more likely to commit crimes later in adulthood. This association remains significant after control of family adversity and drug use. Alcohol and cannabis use during adolescence has no effect on this association. However, main effects of alcohol use on crimes against person and cannabis use on crimes against property are observed. The implications of these findings are discussed.