Natalie Wood and James Dean in Rebel Without a Cause. The film’s director, Nicholas Ray, is said to have slept with both of them. Photograph: Allstar/Warner Bros

David Thomson has spent his life in the dark. Cocooned in the gloom of cinemas, he is licensed to dream with his eyes open. Now, in his late 70s, he is searching for enlightenment, blinking a little as he faces the glare of the reality outside.

Thomson’s criticism – for me the most ingenious and imaginative writing about film – has always been supplemented by cheeky fantasy. In his biographies of Orson Welles and Warren Beatty he overleaps known facts, treating his subjects as characters in a wishful novel of his own; in Suspects (1985) he invents prequels and afterlives for the fatal women and doomed men of film noir. Sleeping With Strangers, however, is the product of a wrenching reappraisal of this daydreaming and the art that encourages it. As the title suggests, films invite us to fall in love with their stars, whom we use as virtual prostitutes. But does this mean that male viewers like Thomson have been trained as voyeurs and potential predators? Could Harvey Weinstein, who produced so many fine films while allegedly ravaging the lives of so many young women, be the bloated, toad-like personification of the art?

Thomson’s publisher originally wanted a history of gay careers in Hollywood, and for a while the book obliges. It playfully examines the cohabitation of buddies who bunk down together in westerns, then decodes some sadomasochistic couplings in film noir – Tony Curtis as Burt Lancaster’s subservient bitch boy in Sweet Smell of Success, Glenn Ford as rough trade picked up George Macready in Gilda. Thomson inconclusively ponders the enigma of Cary Grant’s erotic appeal, and has fun with a dubious tale about Spencer Tracy seducing a handyman he’d summoned to repair his boiler. He also registers appropriate disgust when reporting that, during rehearsals for a play, the supposedly liberal Henry Fonda responded to Charles Laughton’s direction by asking: “What do you know about men, you fat faggot?”

He feels a twinge of compassion for Kevin Spacey, who is “not just ‘out’ now, but erased”

But while Thomson was sifting through all this gossip and innuendo, the ground beneath his feet seismically shifted. Fifty years ago, in his first book, Movie Man, he pointed out that we live in a “visual society”, composed of delectable surfaces; a series of recent scandals has now reminded him that movie men, under cover of their “ill-deserved safety in the dark”, are programmed to look, to lust and probably to assault. The first shock came with Trump’s brazen pussy-grabbing, followed in 2017 by the disgrace of Weinstein. Uncomfortably close to home, the director James Toback, Thomson’s long-time friend, was accused of offering film roles to 38 women in exchange for sexual favours.

In his revulsion, Thomson does not spare himself. A seasoned ogler, he lists the female stars he fantasised about as a teenager, and he incidentally notices that Weinstein’s paltry joy allegedly lay in forcing his victims to watch him masturbate. Have movies “steadily undermined our moral strength”, as boys were once told that “self-abuse” would do? Thomson sadly concludes that “movies have always been a medium for adolescents” – specifically for hormonally jazzed adolescent males.

Despite these misgivings, he is alarmed by the resurgent puritanism to which #MeToo unwittingly caters. He feels a twinge of compassion for Kevin Spacey, who is “not just ‘out’ now, but erased”, and even risks defending the behaviour of the great director Nicholas Ray, who while making Rebel Without a Cause in 1954 slept with Natalie Wood (and also supposedly with its other star, James Dean). Wood was 16 at the time, and had been pimped by her ambitious mother; Ray was 43. Yet could their unlawful intimacy be “a reason why Rebel turned out so well”?

Thomson says Marlene Dietrich, in the films she made for Josef von Sternberg, was “an advertisement for depravity”. Further along, he becomes more anxious, and begins to sound like Plato denouncing the shadows that flicker cinematically in his cave of illusion: “In watching pretence, we acquire... a growing uncertainty over our psychic integrity. What else are movies for?” From then on, the resonant questions accumulate, and we are left to answer them as best we can.

Were the humanists wrong to claim that art is good for us, “like Ovaltine and yoga”? Movies have been more like a secret vice, the first and most invasive of the technologies that have progressively estranged us from one another. As a Londoner resettled in San Francisco, Thomson is also keenly aware of the way the medium has duped Americans, convincing them that happy endings are inevitable so long as gun-toting men control the narrative. “Can’t we admit,” he asks, “how much American experience has been rooted in fear?” In one mind-bending paragraph he outlines an alternative pantheon of great American films – including Citizen Kane, Vertigo and a surprising number of screwball comedies – that expose the country’s gnawing insecurity.

Thomson’s last and boldest speculative forays try to find a way out of the current impasse between the genders, for which he makes movies in large part responsible. Are the sexes doomed to battle for ever, as in Howard Hawks’s His Girl Friday? Did the phallocracy begin to founder when, at the end of Boogie Nights, Mark Wahlberg showed off an absurdly elongated prosthetic penis? Has Meg Ryan’s raucous public orgasm in When Harry Met Sally liberated women, or did it hint that sex has become “an act, a masquerade”? Finally, Thomson wonders if we will ever be able to make love digitally, without needing to touch our fancied partners; to save the race from extinction, he trusts that “some fruitful androgyny” lies ahead.

Either way, his predictions make me want to curl up in the cosy darkness and watch an old movie. But when that alien, artificial future arrives, I hope Thomson will be available to coax me back into the daylight.

•Sleeping With Strangers: How the Movies Shaped Desire by David Thomson is published by Alfred A Knopf (£18.99). To order a copy go to guardianbookshop.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99