It would be impossible to paint “modern life” without touching on the touchy subject of sex.

To the left of the central figure in Édouard Manet’s The Luncheon (1868) there is a black cat, bent over its genitals – assiduously wheedling and scouring with its sandpaper tongue. Easily overlooked, it is a quiet variation on Rembrandt’s niggling, splayed dog in the foreground of his etching Joseph Telling His Dreams. Like Rembrandt, Manet was a realist painter. He was the friend of the realist writers Jules Champfleury and Edmond Duranty – his duel with Duranty notwithstanding. (Manet could be touchy: he publicly slapped Duranty, who had written a review Manet regarded as niggardly.) Zola, the begetter of naturalism, itself an alias of realism, was an indefatigable and trenchant supporter.

This absorbing show at the Royal Academy – composed mainly around Manet’s portraits – has the title “Portraying Life”, which neatly fuses the idea of the portrait with Baudelaire’s crucial coinage, “the painter of modern life”. In general, this entailed the embrace of the contemporary – stovepipe hats, pipeclayed spats – and a rejection of the antique plaster cast, which can stand for the set subjects, the safe syllabus of the academic painter. But in the case of Manet, “modern life” meant something more subtle, more understated than is generally allowed. This exhibition lacks many of Manet’s more notorious paintings, such as, for instance, Olympia, and is therefore a valuable provocation in a different way.

To return to Rembrandt’s etchings, there are several self-portraits in which Rembrandt strives to capture emotion – astonishment, anger, contempt – a little crudely; Sainsbury’s Basics, as it were. Manet’s best portraits are conspicuous refinements, subtly understated, less dramatic, more realistic.

In The Luncheon, Manet gives us a little, implicit, essayistic credo. Evidently set in his studio, to the left is costume bric-a-brac, props – a helmet, two swords, the old way. There is a coil of lemon peel, that standard flourish of expertise. There is an oyster shell – another test of skill – but here without the demanding mother-of-pearl. There is also a benchmark bottle of beer with a cork in it. A background figure is exhaling cigar smoke. At the centre is a young man, the 16-year-old Léon Leenhoff, son of Suzanne, Manet’s wife (née Leenhoff) – and possibly the son of either Manet or of Auguste, Manet’s father. (Suzanne was originally hired as a piano teacher for Manet’s two younger siblings.)

Léon’s enigmatic status is mirrored in his expression, which is often read as haughty but is neutral, occluded, giving nothing away. Its expression is without expression – and utterly convincing, a cul-de-sac of almost intimidating blankness that has us looking elsewhere for clues. At the hand bulging in the pocket of the corduroy trousers, at the straw hat with the black hatband, at the black velvet jacket – all perfectly painted. (Manet is said to have said that all colours existed except for black but he paints it better than anyone. “Black does not exist, that’s the first precept,” reports Gaston La Touche.) As a picture, it is the opposite of Rembrandt and an early marker of modernism’s central inquiry into the actuality of emotions – what we really feel, what we actually express, how much we withhold. It’s modernism as a riposte to romantic overstatement, as an insistence on accuracy. Less is more.

Berthe Morisot with a Bouquet of Violets (1872) is another case in point. She is boldly lit from the left, so half her face is brightly lit, while the other is in pronounced shadow. Again, there are Manet’s incomparable blacks – her piled hat, her scarf, her dress – and her brown hair, her ribbons, so casual, so beautifully natural. It is a portrait of unwavering conviction, from her earrings to the expression in her eyes. What makes this picture so alive? A small thing. A thing you hardly notice. Manet has painted her so that we can tell which is her leading eye. She is looking out of her left eye. It is nothing and it is everything.

Berthe Morisot (1868-69, 1870-71) repeats the casual hair. Her mouth is an inspired daub that brilliantly captures another neglected feeling – preoccupation. Her thoughts are elsewhere. It is a picture that illustrates two characteristics of Manet – the way he draws with paint (like his revered Franz Hals) and the way he trusts to suggestion and avoids the pedantry of finish. Here he uses a starved brush to paint her fur coat and her muff on its strings – differentiating perfectly between the pelts and the muff, the one a species of parquet, the other a big, beautiful burr.

Stéphane Mallarmé (1876) is a tiny masterpiece, showing the poet lolling against a cushion, a cigar in his right hand, the thumb of his left hand outside his jacket pocket. In reproduction, Mallarmé looks, I have always thought, a little bit pissed. The painting puts you right. This is a portrait of someone thinking – and it is all in the eyes once again, which have a look of distance, of inwardness, as they look down to the right. (When we are asked to do mental arithmetic, we look up to the right.) Again, the touch is virtually invisible – especially in reproduction and more so if you are accustomed to Rodin’s Penseur, demonstratively crouched at stool, fist to his forehead, a marble QED.

Another great painting is Portrait of M Brun (1879) – which superficially looks very disappointing. It shows a man with a grey top hat, blue frock coat and white linen trousers. His main features are his watch-chain moustache and button eyes. Hardly anything is happening, it seems. The whole picture is like the last pull of a worn-out plate. And yet Degas bought this from the dealer Ambroise Vollard when Renoir had identified its subject as M Brun. Why? Because it is a painterly feat by a virtuoso. Manet has brought off the impossible. He has painted a recognisable nonentity, a perfect nondescript, a rich nobody, who needed identification.

Interestingly, when Manet embraces finish and larger emotions, it is generally because he is painting thespians, whose I’s are underlined for emphasis. His Portrait of Émilie Ambre as Carmen (1880) is all kiss-curls and costume. Lit from the right, her eyelashes cast a pronounced shadow. The Tragic Actor (Rouvière as Hamlet) (1865) is a painting of acting by an actor. Manet knows they are bigging it up and paints the posturing.

One of the most unfinished portraits here is one of the most successful: Georges Clemenceau (1879-80). It shows Manet’s complete mastery of line and outline. On a background of grey, like a Banksy stencil, Manet lays down the unerring line of the lapel of his frock coat. The head is drawn rather than painted. The outline of the jacket torso is crucially confident. His arms are folded, his hair thinning, his speech on the balcony in front of him. Paul Levy said to me that the work of American artist R B Kitaj derives from this one picture. He is brilliantly right. Much of Toulouse-Lautrec is also implicit in the drawing-painting brushwork of Manet’s The Animal Painter La Rochenoire (1882). Painters owe a lot to Manet, who himself owes much to Velázquez, to Goya, to Ingres, to Hals. And, paradoxically, to the Old Masters, whom he remade.

In this show, we have not Le déjeuner sur l’herbe (1863) but a copy, an artist’s aidememoire, which has the status of a reproduction, useful to Manet perhaps but misleading for us. Manet’s picture, it is well known, is a reworking of a composition detail of Marcantonio Raimondi’s engraving of Raphael’s lost The Judgment of Paris. But what does this enigmatic, scandalous picture mean? How does it relate to modern life? We might begin with the syphilis that caused Manet’s amputated leg and brought about his early death at 51. Or we might begin with the juxtaposition of female nudity and clothed male figures.

In fact, we would be starting in the same place – the brothel. Think about Degas’s monotypes of brothels, where the only clothed woman is the Madame. Manet has cleverly rusticated this topos, blunted the obvious to mute the scandal, but the situation is clear. And the depiction relates directly to the portraits with their understated inflections. The two men take the nudity for granted. They are absorbed in what might be a discussion of philosophy. There is an atmosphere of relaxed gravitas. The woman transfixing the spectator can wait. In the brothel, nudity is ordinary, commonplace, the rule rather than the exception. And if you look at the naked body here, it isn’t sensational. There is no pubic hair, no nipples, no enticement. All the arousal is in her divested clothing spread on the grass. The excited appetite is implied in the overturned basket and its spilled contents, a wicker cornucopia. Which is perfect for the businesslike body of the sex worker before us, patiently waiting.

It would be impossible to paint “modern life” without touching on the touchy subject of sex. Manet’s Olympia (1863) tried the direct address – the barely defiant “so what?” of the courtesan, the sack artist, the cool professional – and ran into even more trouble. Yet these were paintings that referenced a commonplace of masculine life – the prostitute. What about murkier areas?

In his useful study of Manet, Alan Krell writes, “Nothing could be further removed from fancy dress, sexual commerce, and political intrigue than The Railway, the second of Manet’s two works in the Salon of 1874.” I disagree. We are in the realm of sexual commerce. The professional nude model for Olympia, Le déjeuner sur l’herbe, and The Railway was Victorine Meurent. After Manet’s death, fallen on hard times, she wrote to his widow seeking financial assistance. She was also the model for Street Singer (1862), an ambiguous figure on the margin of the demi-monde. With her guitar, the singer has just left a bar and is eating cherries from paper. Her petticoat is visible. Street singer or streetwalker?

Contemporary critics were puzzled by The Railway. Unsurprisingly, because the railway consists largely of background smoke. In the foreground, we have a young girl and Meurent looking straight at the viewer, her face an expressionless mask. In her lap she has a puppy and an unopened fan – both emblems, both clues, both related. Neither, I would suggest, innocent. The fan is waiting to be spread. She also has an open book in her lap. An index finger is keeping her place, inserted into the bare pages.

In MoMA in New York there is a Balthus painting of André Derain. In the background there is a nymphet provocatively raising her leg like Gerty MacDowell arousing Leopold Bloom on the beach in Ulysses. Derain is facing out to the viewer but shows, by a gesture, his awareness of what is behind him. He is wearing a white fly-fronted shirt and is poking his finger into its fabric, which is a synecdoche for a hairless fanny. Might not Meurent’s book be performing the same displaced symbolic function? In Renaissance painting, the same gesture was a demonstration of piety and learning. Typically evoked and mordantly subverted here.

The young girl faces away. She is looking through the railings. She has a big bow, giftwrapping her like a parcel. Her dress is inappropriate for outdoors. Her shoulders are naked. She is wearing earrings. Her hair is coiffed in an adult way. She looks like a grown-up. But her arm has visible puppy fat. To the girl’s right there is a bunch of green grapes resting on a vine leaf . . . I think we are in Jimmy Savile territory, in one of the intractable, unpaintable margins of modern life. Except that Manet has managed to paint it. The girl is for sale. Not so you would notice, unless you were looking. The painting keeps its counsel. It doesn’t denounce or declaim like a Zola. Its careful, realistically concealed innuendo is the merest whisper – audible only if you are listening very, very carefully.

“Manet: Portraying Life” is at the Royal Academy of Arts, London W1, until 14 April