What is it about Hopper? Every once in a while an artist comes along who articulates an experience, not necessarily consciously or willingly, but with such prescience and intensity that the association becomes indelible. He never much liked the idea that his paintings could be pinned down, or that loneliness was his metier, his central theme. "The loneliness thing is overdone," he once told his friend Brian O’Doherty, in one of the very few long interviews to which he submitted.

Why, then, do we persist in ascribing loneliness to his work? The obvious answer is that his paintings tend to be populated by people alone, or in uneasy, uncommunicative groupings of twos and threes, fastened into poses that seem indicative of distress. But there’s something else too; something about the way he contrives his city streets. What Hopper’s urban scenes replicate is one of the central experiences of being lonely: the way a feeling of separation, of being walled off or penned in, combines with a sense of near unbearable exposure.

What Hopper’s urban scenes replicate is one of the central experiences of being lonely: the way a feeling of separation, of being walled off or penned in, combines with a sense of near unbearable exposure

This tension is present in even the most benign of his New York paintings, the ones that testify to a more pleasurable, more equanimous kind of solitude. Morning in a City, say, in which a naked woman stands at a window, holding just a towel, relaxed and at ease with herself, her body composed of lovely flecks of lavender and rose and pale green. The mood is peaceful, and yet the faintest tremor of unease is discernible at the far left of the painting, where the open casement gives way to the buildings beyond, lit by the flannel-pink of a morning sky.

In the tenement opposite there are three more windows, their green blinds half-drawn, their interiors rough squares of total black. If windows are to be thought analogous to eyes, as both etymology, wind-eye, and function suggests, then there exists around this blockage, this plug of paint, an uncertainty about being seen – looked over, maybe; but maybe also overlooked, as in ignored, unseen, unregarded, undesired.

In the sinister Night Windows, these worries bloom into acute disquiet. The painting centres on the upper portion of a building, with three apertures, three slits, giving into a lighted chamber. At the first window a curtain billows outward, and in the second a woman in a pinkish slip bends over a green carpet, her haunches taut. In the third, a lamp is glowing through a layer of fabric, though what it actually looks like is a wall of flames.

There’s something odd, too, about the vantage point. It’s clearly from above – we see the floor, not the ceiling – but the windows are on at least the second storey, making it seem as if whoever’s doing the looking is hanging suspended in the air. The more likely answer is that they’re stealing a glimpse from the window of the ‘El’, the elevated train, which Hopper liked to ride at night, armed with his pads, his fabricated chalk, gazing avidly through the glass for instances of brightness, moments that fix, unfinished, in the mind’s eye. Either way, the viewer – me, I mean, or you – has been co-opted into an estranging act. Privacy has been breached, but it doesn’t make the woman any less alone, exposed in her burning chamber.

‘City of glass’

This is the thing about cities, the way that even indoors you’re always at the mercy of a stranger’s gaze. Wherever I went – pacing back and forth between the bed and couch; roaming into the kitchen to regard the abandoned boxes of ice cream in the freezer – I could be seen by the people who lived in the Arlington, the vast Queen Anne co-op that dominated the view, its 10 brick storeys lagged in scaffolding. At the same time, I could also play the watcher, Rear Window-style, peering in on dozens of people with whom I’d never exchange a word, all of them engrossed in the small intimacies of the day. Loading a dishwasher naked; tapping in on heels to cook the children’s supper.

This is the thing about cities, the way that even indoors you’re always at the mercy of a stranger’s gaze

Under normal circumstances, I don’t suppose any of this would have provoked more than idle curiosity, but that autumn wasn’t normal. Almost as soon as I arrived, I was aware of a gathering anxiety around the question of visibility. I wanted to be seen, taken in and accepted, the way one is by a lover’s approving gaze. At the same time I felt dangerously exposed, wary of judgement, particularly in situations where being alone felt awkward or wrong, where I was surrounded by couples or groups. While these feelings were undoubtedly heightened by the fact that I was living in New York for the first time – that city of glass, of roving eyes – they arose out of loneliness, which agitates always in two directions, towards intimacy and away from threat.

That autumn, I kept coming back to Hopper’s images, drawn to them as if they were blueprints and I was a prisoner; as if they contained some vital clue about my state. Though I went with my eyes over dozens of rooms, I always returned to the same place: to the New York diner of Nighthawks, a painting that Joyce Carol Oates once described as “our most poignant, ceaselessly replicated romantic image of American loneliness”.

I don’t suppose there are many people in the western world who haven’t peered into the cool green icebox of that painting, who haven’t seen a grimy reproduction hanging in a doctor’s waiting room or office hallway. It’s been disseminated with such profligacy that it has long since acquired the patina that afflicts all too-familiar objects, like dirt over a lens, and yet it retains its eerie power, its potency.

I’d been looking at it on laptop screens for years before I finally saw it in person, at the Whitney one sweltering October afternoon. Up close, the painting rearranged itself, decomposing into snags and anomalies I’d never seen before. The bright triangle of the diner’s ceiling was cracking. A long drip of yellow ran between the coffee urns. The paint was applied very thinly, not quite covering the linen ground, so that the surface was breached by a profusion of barely visible white pinpricks and tiny white threads.

Green shadows were falling in spikes and diamonds on the sidewalk. There is no colour in existence that so powerfully communicates urban alienation, the atomisation of human beings inside the edifices they create, as this noxious pallid green, which only came into being with the advent of electricity, and which is inextricably associated with the nocturnal city, the city of glass towers, of empty illuminated offices and neon signs.

The diner was a place of refuge, absolutely, but there was no visible entrance, no way to get in or out. There was a cartoonish, ochre-coloured door at the back of the painting, leading perhaps into a grimy kitchen. But from the street, the room was sealed: an urban aquarium, a glass cell. Inside, in their livid yellow prison, were the four famous figures. A spivvy couple, a counter-boy in a white uniform, his blond hair raked into a cap, and a man sitting with his back to the window, the open crescent of his jacket pocket the darkest point on the canvas. No one was talking. No one was looking at anyone else. Was the diner a refuge for the isolated, a place of succour, or did it serve to illustrate the disconnection that proliferates in cities? The painting’s brilliance derived from its instability, its refusal to commit.

Was the diner a refuge for the isolated, a place of succour, or did it serve to illustrate the disconnection that proliferates in cities?

Look, for instance, at the counter-boy, his face maybe affable, maybe cold. He stands at the centre of a series of triangles, presiding over the nocturnal sacrament of coffee. But isn’t he also trapped? One of the vertices is cut off by the edge of the canvas, but surely it’s narrowing too sharply, leaving no room for the expected hatch or gangway. This is the kind of subtle geometric disturbance that Hopper was so skilled at, and which he used to kindle emotion in the viewer, to produce feelings of entrapment and wariness, of profound unease.

What else? I leant against the wall, sweaty in my sandals, itemising the diner’s contents. Three white coffee cups, two empty glasses rimmed in blue, two napkin dispensers, three salt shakers, one pepper shaker, maybe sugar, maybe ketchup. Yellow light flaring on the ceiling. Livid green tiles (brilliant streak of jade green, Hopper’s wife Jo had written in the notebook she used to log his paintings), triangular shadows dropping lightly everywhere, the colour of a dollar bill. A hoarding above the diner for Phillies American cigars, Only 5cs, illustrated with a crude brown doodle. A green till in the window of the store across the street, not that there was any stock on show. Green on green, glass on glass, a mood that expanded the longer I lingered, breeding disquiet.

The window was the weirdest thing: a bubble of glass that separated the diner from the street, curving sinuously back against itself. This window is unique in Hopper’s work. Though he painted hundreds, maybe thousands, in his life, the rest are simply openings, apertures for the eye to gaze through. Some catch reflections, but this was the only time he ever painted glass itself, in all its ambiguous physicality.

Simultaneously solid and transparent, material and ephemeral, it brings together what he elsewhere did in parts, fusing in one devastating symbol the twin mechanisms of confinement and exposure. It was impossible to gaze through into the diner’s luminous interior without experiencing a swift apprehension of loneliness, of how it might feel to be shut out, standing alone in the cooling air.

This is an edited extract from The Lonely City by Olivia Laing, published this week by Canongate Books. Olivia will be interviewed at the Barbican, London on 8 March by Andrew Graham-Dixon.

If you would like to comment on this story or anything else you have seen on BBC Culture, head over to our Facebook page or message us onTwitter.