Joachim Lafosse’s Private Property, from a screenplay by Mr. Lafosse and François Pirot (in French with English subtitles) fades out with an end-scrawled mocking dedication to “boundaries,” of which there are precious few in the peculiar family circle of Pascale, a divorced Belgian-French mother (Isabelle Huppert), and her two overgrown fraternal-twin sons, Thierry and François (played by Jérémie and Yannick Renier, who are brothers in real life). The sheer ridiculousness of the situation would be ideal for a naggingly domestic farce about one’s children and their refusal to grow up and assume their adult responsibilities. But Ms. Huppert and the two Reniers play their parts with deadpan and, in the end, almost deadly seriousness. Pascale cooks, washes and irons for the two boys while they literally sprawl around the oversize farmhouse purchased for them and Pascale by their indulgent father Luc (Patrick Descamps), who lives nearby with a new wife and child. Luc occasionally drops by to give them (and Pascale) little bundles of ready cash, much to his ex-wife’s noisy displeasure. Pascale seems actually jealous of Luc’s perceived attempts to curry favor with his good-for-nothing sons.

Eventually, Private Property becomes the kind of film that creeps up on you as you are experiencing it, ever-watchful for explosions of sheer exasperation on Pascale’s part—explosions that never come. This makes Pascale yet another choice role for Ms. Huppert, a hypnotically controlled actress who has become more implacably mysterious as she has gotten older. Her characters never plead for our sympathy or understanding: Pascale seems monstrously impervious to the most brazen assaults on her dignity and self-respect, though the actress somehow convinces us that she retains an inner reserve of indomitable self-awareness. Ms. Huppert has steadily attracted attention in the almost 60 screen appearances since she began her career in 1971. I raved about her in Maurice Pialat’s Loulou (1980), right after she had survived Michael Cimino’s disastrous Heaven’s Gate the same year. Over the years, she has been one of Claude Chabrol’s favorite actresses, and she remains one of the most sensual and erotic presences in the cinema, despite a persistent inscrutability of expression.

The scene that everyone is talking about in Private Property involves Pascale and Thierry standing in a small bathroom with no shower curtain while the mother takes a shower and Thierry brushes his teeth before a mirror reflecting his mother’s naked image. The matter-of-fact manner in which Mr. Lafosse photographs the spectacle and the detached characters within it increases the fearful tension for the audience.

The plot thickens when Pascale begins primping herself up for a date. Thierry teases her outrageously by suggesting that all her prettying-up has made her look like a slut, and that her new dress would look better on François. For the most part, François laughs along with Thierry, until he feels that his more aggressive brother is going too far. Throughout all the teasing by both boys, Pascale remains miraculously unperturbed. She seems aware, however, that Thierry is the bigger problem. Indeed, Thierry relishes his special stature by taunting François for being a mama’s boy, and he keeps blaming Pascale for the divorce.

Hence, when Pascale begins hinting that she would like to quit her job (which is never shown, described or even identified), sell the house and open a bed-and-breakfast somewhere nearby, it is Thierry who explodes with rage by claiming that the father bought the house for his two sons, and that his mother has no right to sell it.

Meanwhile, Pascale’s new boyfriend Jan (Kris Cuppens) pressures her to leave her sons and start the bed-and-breakfast with him. But after Pascale consents to a meeting among everyone concerned at the farmhouse, a stormy confrontation ensues between Jan and Thierry—while Pascale sits quietly with her maddening passivity. Jan quickly realizes that Pascale cannot bring herself to have it out with Thierry, and tells her flatly that only she can break the impasse. This she seems disinclined to do, and so Jan departs from her life.

Art-film connoisseurs may remember having seen Jérémie Rainier growing up in more sympathetic roles for the Dardenne Brothers in La Promesse (1996) and L’Enfant (2005). Here, he brings new dimension to the near-fatal consequences of family life without boundaries for its incorrigibly immature members. Only in the last receding, traveling shot do we come to realize how big the farmhouse that has claustrophobically enclosed Pascale, Thierry and François for most of the film really is, and how far it is, literally and figuratively, from reality and civilization. Mr. Lafosse has made a mordant movie beyond genres—and one that is too mesmerizing to miss.