Second place together ... Steve Carell and Channing Tatum in Foxcatcher

Bennett Miller's sports movie Foxcatcher – based on a grisly true story – is a superb tragicomedy of the beta-male, a nightmare of the also-ran and almost-ran. It is also a deeply strange story about a strange man whose insecurities were all too ordinary and explicable.

The movie is like an agonising drama based on Jerry Seinfeld's routine about the horror of the silver medal. ("Silver? You're the No 1 loser.") And above everything else, it is a piercing insight into toxic mentor-ism, into competitive men and their terrible emotional need to find a father-figure to hate and to disappoint.

Foxcatcher Farm is the magnificent Pennsylvania home of the DuPont family, a colossal mansion created in imitation of the English country house: its super-rich Wasp inhabitants evidently affected to admire the English tradition of riding to hounds – hence the name. In the mid-80s, it was home to John DuPont – played here by Steve Carell with a superb feel for the role's absurdity and anguish, but stopping short of caricature. He is a lonely, prickly and arrogant plutocrat who resembles a middle-aged spoilt child. He has had years to develop a sense of entitlement and the rich man's capricious and demanding manner. Suspecting that he is a disappointment to his cantankerous patrician mother (an exquisite cameo from Vanessa Redgrave) and suspecting at a deeper unconscious level that inheriting her wealth is his only achievement, John sets out to achieve something: he decides to bankroll America's national wrestling team and use his staggering wealth – and, of course, his visionary patriotism and leadership – to win Olympic gold for the US at the 1988 Seoul Games.

So John gets in touch with a blue-collar guy called Mark Schultz (Channing Tatum), a wrestler and gold-medal-winner at the '84 LA Games, who is the younger brother of the smarter, more charismatic and more successful wrestler and coach Dave Schultz (Mark Ruffalo). Poor Mark has always been second banana to Dave, who is nonetheless affectionate and protective about his great lunk of a brother. Mark is used to people talking to him when they really want to talk to his cooler brother. So he is astonished and then profoundly thrilled to be called up one day by the eccentric and sinister DuPont and invited to live and train at Foxcatcher, and become, with him, a founding father of American wrestling greatness.

But wait. Why did DuPont seek him out and not Dave? DuPont mouths tender platitudes about seeing Mark's potential and allowing him to emerge from his brother's shadow. But clearly getting Mark on board was at some level just a ploy to get him to persuade Dave to join them both at Foxcatcher – a ploy that finally succeeds. Yet Miller and screenwriter E Max Frye brilliantly allow you to see that it is more complicated: DuPont felt it easier to boss Mark around, and also felt an awful sense of identification with this fellow second-class male. Whatever the reason, Mark is angry and humiliated when he realises his new, rich quasi-dad prefers Dave; Dave, for his part, is angry and humiliated at having to kowtow to his new master. And then DuPont himself is angry and humiliated to realise that both of these guys – these real men whose self-respect he has attempted to buy – despise him. The scene is set for a disaster.

Just as in his previous film, Moneyball (2011), Miller here reveals a connoisseur's eye for the arenas and corporate spaces of sport: the training rooms, gyms, pennants, the offices – all the exciting outward show of sports prestige. Miller has a lovely scene when DuPont shows Mark the sumptuously appointed new training area he has set up at Foxcatcher, and with boyish excitement, Mark can't help going into some moves, trying it out.

The act of wrestling itself is an absorbing and obviously metaphorical contest. It is an unerotic clinch: as intimate as dancing. In an early training encounter, Mark is clearly furious with himself for being bested by Dave and accidentally-on-purpose butts him in the face. Dave just wipes away the blood and carries on. He doesn't say anything and his manly reticence and forgiveness just makes Mark's sense of defeat worse.

It is a trio of wonderful performances. Channing Tatum's Mark is vulnerable and sad; Mark Ruffalo's Dave is smart and professional and his shame at taking the DuPont shilling is correspondingly intense. And Steve Carell's DuPont is a compelling monster – but a monster who inspires not fear but pity.

Perhaps Foxcatcher may yet create a new genre to match the boxing movie: the "wrestling movie" – the kind that Barton Fink was trying to write in the Coens' film. Certainly Mark's rage in his hotel room at losing – smashing his head against the mirror and then eating himself into a porky stupor – must have been inspired by De Niro's Jake LaMotta.

It is a gripping film: horrible, scary and desperately sad.

• Comments have been reopened to time with this film's Australian release