How serious is John le Carre? There is a feeling among his many admirers that he is very serious indeed, not just an accomplished genre writer, but more than that: the natural heir of Joseph Conrad and Graham Greene, a writer whose worldliness, political acumen and commanding interest in the defining moments of contemporary history have resulted in a postwar body of work of unrivalled political complexity. But le Carre himself, you suspect, has long felt undervalued by what he calls the “literary bureaucracy” - by which he means the coteries of critics, career novelists, agents and publishers who gather at the same London parties and events. “If you move in these circles,” le Carre once said, “you trip over connections at every point . . . I don’t know the people who review me, I don’t go to their parties - I never will. I have the most profound contempt for the system - a total alienation from it.”

David Cornwell (le Carre was a pseudonym to preserve his diplomatic cover), now 67 years old, removed himself early in his writing life from this airless world, when the international success of his third (and wondrously plotted) novel, The Spy Who Came in From the Cold (1963), eased his departure from the British “intelligence” service into which he had stumbled as a student linguist. Since then he has lived for most of the time in Cornwall, while keeping a house in Hampstead, a self-styled outsider.

Yet, in many ways, le Carre, as a former diplomat and servant of MI5 and MI6, has been at the centre of conventional society; and indeed he can be a conventional writer, locking himself in the prison of the genre of the spy novel, no matter how much he attempts to stretch and bend the bars that too often constrain him. His new novel, Single & Single (Hodder & Stoughton, £16.99), about the intrigues of the bandit capitalists of the new Russia, displays his obvious weaknesses: the flat, inexact dialogue, the unhappy flirtation with cliche (people drift down “memory lane”) the perfunctory description, the febrile, over-elaborate plotting, the inevitable certainty of closure.

And yet his books remain hard to forget long after you have finished them. This has something to do, I think, with his profound understanding of, and engagement with, the political world in which he grew up. From the beginning, he had an urgent subject - the cold war - and a compelling preoccupation - secrecy. As a novelist, le Carre is obsessed with secrecy, as Conrad was, secrecy as a way of life and as an extended metaphor through which to understand human motivation (public and personal betrayal are inextricably bound up in his novels, as the cuckold George Smiley realises whenever he contemplates the sadness of his marriage).

Le Carre understands that there is no one lonelier than the double agent: addicted to duplicity and loyal only to himself, he lives in a condition of acute watchfulness. His fiction, with its suspensions, narrative absences and aporias, leaves much unsaid. Even when his novels reach their inevitable resolution, as the genre demands, there is nevertheless a powerful sense of incompleteness, of uncertainty and confusion, as though the spooks themselves are unable to comprehend the events that have just passed, or indeed the value of what they are working for or against. So always in le Carre demystification leads to a greater mystification.

If le Carre is to be believed, he did not have to search long to find his subject; the secret condition, as he points out, “was imposed on me by birth, under the influence of that monstrous father. Then that brief passage through the secret world sort of institutionalised it.” That monstrous father is Ronnie Cornwell, an inveterate conman and recidivist who spent several terms in prison and about whom le Carre wrote so memorably in A Perfect Spy - described by Philip Roth as the most accomplished British novel since the war. Such a claim is not as absurd as it might sound. A Perfect Spy functions on many levels: as a thriller, as a complex family history, as a study in the unreliability of memory, as an exercise in multiple narratives and time shifts; and as a metaphysical quest narrative, where the actual search for a missing spy, Magnus Pym, is mimicked on a more local level by Pym’s own internal search for the deceitful father whom he thinks he hates but never really knew. In every way, it is exceptional.

There is something appealingly complex in le Carre’s withdrawal from fashionable society, in his fondness for casting himself in the role of elevated outsider. To meet him is to meet, on first impression, a lifelong member of the professional middle-upper classes, a tall, handsome, urbane member of the Oxbridge elite. It is a false impression. “I just think that it’s a part I put on,” he has said. “It never occurs to me that people could imagine I was well born, or that I was secure in the company of the British establishment, because it really isn’t so. I mean, I’ve shafted it for as long as I’ve been writing.”

So there you have it: le Carre as the loner attracted to labyrinthine institutions and secret conclaves yet paradoxically working at the same time to subvert them; the former Eton schoolmaster and MI6 officer who purports to loathe the social prejudices and class structures of English life; the multi-millionaire, hawkish cold war propagandist who claims to be a man of the left and to despise the “ever-growing gap between the very rich and very poor” in Britain. All these careful contradictions, along with his chaotic, itinerant childhood (Ronnie was always defaulting on his son’s school fees), have made le Carre one of the most gossiped-about writers in England. As a result, he has withdrawn even further.

“I don’t think David is secretive in a bad way,” says his former agent George Greenfield, through whom le Carre met his second wife, Jane. “I agree he is very elusive and doesn’t like to give anything away, particularly to interviewers; but there’s more of him in A Perfect Spy than in any other of his books. It’s all there.”

Yet, like the spook he once was, le Carre occasionally breaks cover to thwart unwanted interest into his private affairs. This happened when the journalist Graham Lord’s confidential synopsis of his proposed biography of le Carre, which had been circulated to publishers, was leaked. “I was served with a writ for libel by David,” Lord tells me. “It was a very uncomfortable experience to be pursued by a very rich man for libel, especially as at the time I was suing the Express for constructive dismissal and my ex-wife was demanding more alimony. I had too many lawyers after me. I concede that there was material in the synopsis that was defamatory and would have needed a great deal of checking; but it wasn’t meant for publication.” As part of the eventual settlement, Lord agreed not to write his unauthorised biography (Robert Harris is thought to be working on the authorised one).

Le Carre, too, is not averse to provocation. And he never forgets a slight, which may account for his feud with Salman Rushdie, which stretches back to the late 1980s, when Rushdie wrote a sneering review of The Russia House, le Carre’s novel about the early years of perestroika. “John le Carre,” he wrote, “wants his work to transcend the genre and be treated as Serious Literature . . . [But] much of the trouble is, I’m afraid, literary. There is something unavoidably stick-like about le Carre’s attempts at characterisation.”

Le Carre was later accused of being “unfeeling for Salman’s position” following the Ayatollah Khomeini’s fatwa against the author of The Satanic Verses. What he had said was that someone of Rushdie’s background “made light of the Book at your peril . . . A peculiar justification used by Rushdie’s most vociferous defenders is that his novel has great literary merit - some insist it is a masterpiece . . . Are we to believe that those who write literature have a greater right to free speech than those who write pulp?” Most recently, in an exchange of letters in the Guardian last year, Rushdie called le Carre “an illiterate pompous ass”; and le Carre in turn ridiculed Rushdie as “a self-canonising, arrogant colonialist”.

What is interesting about the feud is not only what it reveals about le Carre’s motivation and sense of self-worth, but also how vividly it dramatises the disjunction in English fiction between the literary and popular novel, a disjunction originating in the mould-breaking modernism of Joyce, Eliot and Pound, and in their contempt for generic repetition and established forms.

But le Carre is not the literary barbarian that Rushdie and his supporters would have it. In The Secret Pilgrim (1990), his requiem for the cold war, an aged George Smiley warns, while addressing an audience of young recruits, that Russia can never be trusted. “For one reason, the Bear doesn’t trust himself. The Bear is threatened and the Bear is frightened and is falling apart . . . The Bear is broke, lazy, volatile, incompetent, slippery, dangerously armed . . . “

Conrad said that he wrote Under Western Eyes (1911), his great novel of espionage and betrayal set in pre-revolutionary Russia, “to render not so much the political style as the psychology of Russia itself”. Conrad’s Russia is a “monstrous blank page awaiting the record of an inconceivable history”. It is a country on the edge of complete moral anarchy. Le Carre has provided, too, a valuable psychological record of Russia’s inconceivable post-revolutionary history under communism. That is his considerable, fittingly contemporary achievement; and that is why I suspect his work will enjoy a long afterlife as future generations turn to him to discover how it was to live through the long, anxious years of the cold war. Through reading him, too, they will better understand something of the failure and complacency of the English elite that once sought to rule the world but ended up unable to preserve the unity of even the British union itself.