How should we read psychoanalysis? Many of its great theorists – Sigmund Freud, Donald Winnicott, Jacques Lacan – trained as doctors, and their successors tend to follow the rigid formulae of academic papers. However, for Adam Phillips, a practising psychoanalyst who is also a perceptive literary critic, it is "more illuminating" to consider psychoanalysts as poets "rather than failed or aspiring scientists".

This attitude is characteristic of how, since the publication of On Kissing, Tickling and Being Bored 20 years ago, Phillips has distinguished himself from most psychoanalytical writing. He is fond of playfully addressing subjects (freedom, boredom) that he feels his colleagues have neglected, and in a form – the literary essay – far removed from clinical studies. After 17 books, this collection brings together pieces that span Phillips's career, arranged chronologically, although it excludes his writing on literature.

Apart from an ill-fitting essay on the photographer Diane Arbus, the focus is on psychoanalysis: its history, methods and above all its preoccupations, among them desire, memory and narcissism. The book is crammed with quotes and references that Phillips often employs ingeniously. In The Dream Horizon, he compares Giorgio Vasari's writing about art in the 16th century to the way we talk about dreams, "offering strange descriptions to evoke the character of a work not known by the listener". His prose is always elegant – if occasionally long-winded – and throws up striking phrases: the bored child is described as "a sprawl of absent possibilities".

Phillips is devoted to Freud and is a lucid expositor of his ideas, especially in The Master-Mind Lectures. He does not put forward theories of his own, though, and for all his flashiness and erudition, there is something modest about his approach. He is interested in some of the more mundane areas of human behaviour and accepts the limits of his understanding: "the only thing the psychoanalyst can't afford to do," he writes, "is to have too much of a sense that he knows what he is doing."

The closest Phillips comes to prescription – or even self-help – is urging modesty on his readers. We need not, he insists, strive to know ourselves completely, nor be "too convinced of our pleasures" when we wish for things. Instead, we should submit to chance and live as if "our lives are subject to accident". Phillips thinks we make too much of happiness; he is a champion of frustration, which is his great subject. In Punishing Parents, a recent essay that is one of the book's highlights, he writes that the parent who punishes a child's tantrum teaches him or her that "rage and frustration create nothing but rage and frustration", when they should be making that frustration more bearable.

Reading Phillips can itself be a frustrating experience. His work is dotted with tantalising questions that are never answered. He might end an essay with one of these, or finish abruptly on an aphorism – "our eagerness for repetition can be self-blinding" – just when you think a riddle is about to be solved. Writing, for Phillips, seems to be a kind of improvisation in which, as he has said, "things are being worked out, not resolved". He repeatedly considers ideas or phrases from a different angle – the "another" way of the book's title – and teases out new meanings, sometimes brilliantly.

Such lively intelligence wins over the reader and makes Phillips's work addictive. It is best to approach it as he urges us to read psychoanalysts: rather than worrying about whether they're right, "we can just argue instead about whether their words are persuasive, eloquent, evocative or beautiful". On those counts he mostly succeeds. You won't mistake him for a scientist.