This book is something you don't see every day: a novel steeped in both high modernism and continental philosophy that's being rolled out as a publishing event in the UK and US. Tom McCarthy, its author, is a 41-year-old Londoner who went to Dulwich College and studied English at Oxford when the literary theory boom was at its height. After spending time in Prague and Amsterdam, he surfaced in 1999 as the general secretary of the International Necronautical Society, a semi-fictitious avant-garde group co-masterminded by the philosopher Simon Critchley, and began to stage events at such venues as the ICA. His first novel, Remainder (2005), later described by Zadie Smith as "one of the great English novels of the past 10 years", was originally put out by a Paris-based art publisher, and though another novel, Men in Space (2007), and a book on Tintin soon followed, he was more of a figure on the gallery circuit than in the literary world until Remainder's reputation began to mushroom.

In articles, lectures and interviews, McCarthy speaks the language of post-humanism. His allegiance is to James Joyce and Samuel Beckett, the French nouveau roman and post-structuralist modes of thought; with a few exceptions, such as William Burroughs and Thomas Pynchon, most English-language writing since modernism's heyday can be written off as naive, reactionary stuff. It's bracing and fun to see these views being aired in a stubbornly non-modernistic literary culture. But McCarthy's art world affiliations, and the rather arts-institutional intellectual currency he trades in, also raise the suspicion that his end product might turn out to be a bit pretentious, in the style of Deleuze-loving architecture theorists or Lacan-quoting gallery notes. This suspicion isn't totally off the mark, yet McCarthy is a talented and intelligent novelist; however pretension-prone the scene he's interested in might be, his writing is tight and lucid, and he has a functioning sense of humour.

C is a 1960s-style anti-novel that's fundamentally hostile to the notion of character and dramatises, or encodes, a set of ideas concerning subjectivity. On the face of it, though, it's a historical fantasy, sometimes witty and sometimes eerie, built around the early years of radio transmission. The central figure, Serge Carrefax, is born in 1898 on an estate named Versoie in southern England. His father, an eccentric inventor, oversees a school for deaf children; his mother, who is deaf and was once the father's pupil, manufactures silk. Serge and his older sister, Sophie, grow up surrounded by transmitters and insects; Serge gets the wireless bug, while Sophie develops an interest in natural history. Time passes, punctuated by their father's elaborate school plays, pageants based on Elizabethan translations of Ovid. Then, after her initiation into adult sexuality, Sophie starts channelling cryptic messages. With the first world war looming, she drinks a glass of cyanide.

Sophie's death and interment hang heavily over Serge's subsequent career in a way that's far from being conventionally novelistic. Though he's sent to a spa town in central Europe to be treated for "black bile", Serge doesn't do much in the way of emoting, being more interested in precise spatial perceptions and the feelings he gets from dialling through radio frequencies. Under the influence of his godfather, a jovially sinister cryptographer named Widsun, he heads off to the war as a wireless operator in spotter planes over the front – an experience he enjoys in a Futurist kind of way. Having acquired a taste for cocaine and heroin, he turns up next in interwar London, studying architecture and tangling with flappers and fraudulent spiritualists. Finally, in 1922 (a key year in the history of literary modernism), he's sent to Egypt to help set up a world-spanning imperial communications network, a task that takes him to an archaeological dig where McCarthy dispenses a few of the keys to what is, by this stage, an immense symbolic superstructure.

Needless to say, Serge isn't a rounded character. He himself has trouble getting to grips with perspective; at one point someone studies his features "as though trying to draw their flat inscrutability out into some kind of relief". Like the narrator of Remainder, he's projected as a blank everyman, with a blokey, quizzical attitude to high-flown statements. But while he isn't an arty or intellectual figure, everything around him bursts with both qualities, from the novel's multivalently punning nomenclature to the micro-organised threads of imagery and argument involving Greek myth, Renaissance verse, geometry, earth and insects (these last, as in Finnegans Wake, playing on "incest"). Though Serge holds the foreground, it's plain from early on that the novel is chiefly structured by the idea of transmission and reception, which serves as a metaphor for, among many other things, and very roughly speaking, an implied relationship between language, technology and subjectivity.

The near-Joycean scale and density of all this is truly impressive, as is McCarthy's ability to fold it into a cleanly constructed narrative, which has its boring stretches but also moments of humour and weird beauty. Yet its mind-blowingness as a reading experience depends on the reader's appetite for certain types of analysis. Armed with various concepts from Heidegger, Freud or Paul Virilio, say, it would be possible to unpick its implications more or less indefinitely, but there's a dispiriting feeling that the book has been reverse-engineered with an eye to achieving just that. On the other hand, Sophie's death, which is partly an allegory for lost philosophical certainties, can also be read as taking on an emotional weight that goes against the grain of the novel's ostensible scorn for squishy psychologising. "Will he turn out," McCarthy asked recently of the French writer Jean-Philippe Toussaint, "to have been deconstructing literary sentimentalism or sentimentalising literary deconstruction?" It's a sign of his writerly horse sense that this skilfully realised, ambitious, over-literary book finds the time to leave a similar question hanging.