Long before the devastating attacks on New York and Washington, America seemed haunted by a strange and persistent melancholy, even in the midst of spectacular superabundance. Searching for secular redemption, modern Americans long to become engines of their own self-creation, freed from any taint of the past. But Americans are prisoners of their own achievement, never able to reach the limits of their ever-proliferating desires. What remains is hideous boredom.

Modern America, for all its professed religiosity, is in thrall to a peculiar nihilism; it is essentially an entertainment culture, addicted to narratives of catastrophe. American video games, disaster movies and concept thrillers have long been sustained by eschatological anxiety. From an early age, children are taught to fear the end of America itself, the destruction by malign forces of all that is held most true. Once, these forces came from outer space, later, from behind the Iron Curtain, carrying an atheistic date stamp. Today, they come from the Islamic world - and they have never been more threatening.

No one exploits American fears of, and unconscious longing for, catastrophe more expertly than the concept-thriller writer Tom Clancy, said to be the highest-paid author in the world (his recent two-book deal is worth a reported $45m). Clancy is a cultural phenomenon: arguably the most popular novelist on earth, with more than 30 million of his books in print, an unreconstructed Republican hawk and a close friend of the Reagan family and Oliver North. “If you don’t like driving a tank, there’s something wrong with you,” Clancy once said. His work is saturated with research and hard detail; his descriptions of nuclear submarines and fighter jets have a startling authenticity—the boffin porn of the teenage defence electronics fanatic. “A lot of what I know about warfare I learnt from reading Tom,” said Colin Powell, the US secretary of state and former chairman of the joint chiefs of staff.

To read Clancy, then, is to understand why so many Americans wish to withdraw behind a virtual defensive shield, leaving themselves free to wander overlit shopping malls of limitless mediocrity, in isolation from a crazy world. In Clancy’s books, the world is always close to or at war and the US is threatened with extinction. “Is this the promised end?” asks Kent in King Lear. “Or,” replies Edgar, “image of that horror?” Through reading Clancy, Americans have lived vicariously with a sense of an ending, simultaneously embracing what they most fear and perhaps most desire - the ruin of cities, the collapse of nations, the vanquishing of alien peoples. Clancy may not yet have shown them the promised end (though he has come close), but he has repeatedly shown them images of that horror: assassination and the collapse of skyscrapers; battles at sea, on land and in the sky; perpetual conflict in the Middle East; and, most poignantly, the hijacking by Arab militants of civilian planes so that they may be used as le thal weapons against the American people (The Sum of All Fears, 1991). These images of horror have been replicated endlessly in Hollywood movies and in computer games, so that it is no exaggeration to describe Clancy as the novelist who comes closest to understanding and animating the modern American psyche: paranoid, deluded, isolated and aggressively confrontational.

Clancy has become his own global brand. He has franchised out his name to a team of ghost-writers who produce, under the “Op-Center” consortium, books that are bestsellers but have none of the lustre or huge narrative momentum of his own techno-thrillers. Hollywood agents scramble to buy the multimillion-dollar film rights to his books, including Patriot Games, which was about a plot by the IRA to murder the British royal family and starred Harrison Ford as Clancy’s all-action hero Jack Ryan; and The Hunt for Red October, which featured Sean Connery in a story about a rogue Soviet nuclear submarine. Clancy also owns Red Storm Entertainment, a software company that creates video games from the material of his fiction and which, he believes, will enable him to create a “new art form”. “Instead of telling them to people as you do if you’re a playwright or an author, we present the reader with stories in which he can participate.”

They have right on their side, and if you are not for them, you are against them. This is a fictional world of simple and incisive oppositions, a binary realm of good and bad, black and white, right and wrong.

Acute myopia prevented Clancy from joining the armed forces. The wound still fester s and perhaps explains why he is often photographed in military clothing and why he parked a tank on the front lawn of his 80-acre Maryland estate. “I wanted to serve my couple of years as a lieutenant,” he said. “I just thought I owed that to my country. But they didn’t want me; that’s the name of the game.” The heroes of his novels are men (they are always men) of action, monoliths of courage and self-affirmation.

The son of a postman from Baltimore, Clancy was working as a door-to-door insurance salesman when he began work on his first novel. It was not published until 1985, when he was 38, and then only quietly, without marketing or advertising. Yet its analysis of cold-war geopolitics and portrayal of nuclear sub marine technology was so accurate as to prompt accusations that he had seen classified documents. Within a year, he was a millionaire, delivering lectures to the US National War College and meeting regularly at the White House with Ronald Reagan, still described by Clancy as “America’s greatest president”.

In person, Clancy is big, swaggering, deep-voiced and straight-backed—the John Wayne of fiction. He speaks in rapid, staccato sentences, studded with aphorisms and gnomic utterances. With his thick, dark, steel-framed spectacles, he resembles a highway cop, or perhaps a hitman. The blue smoke from his constant cigarette spirals and curls.

I first met him at the National Liberal Club in Whitehall, when he was visiting London to address the monthly forum of senior naval staff at the Ministry of Defence. Dressed in their lounge suits, the men from the Royal Navy filed sombrely into the club, huddling together beneath a bust of Lloyd George. It was a perfectly English occasion, whisperingly conspiratorial and clubbable. Then Clancy strode into the room, accompanied by an alarmingly tall blonde woman whose big, buoyant hair had all the elaborate intricacy of a wasp’s nest (Clancy had recently separated from his wife of more than 20 years). The mood was transformed. Captain Peter Hoare, the head of defence studies in the Royal Navy, introduced Clancy by saying that he needed “no introduction”. Clancy nodded approvingly, stubbed out his cigarette and, to the accompaniment of polite laughter, said: “I’ve never been in a liberal club before. I’m a conservative.”

From there, speaking without notes, he delivered his theory of warfare, steeped in anti-communist and anti-Islamic sentiment. He roamed restlessly across centuries and offered potted histories of the great battles in an engaging, wised-up vernacular. It was an aggressive, bravura performance that left much of his audience shifting with patrician unease. Although the forum was private and nothing was to be reported, a worried Captain Hoare rang me the next day. He wanted to reiterate that nothing Clancy had said was “official naval policy; I thought he was going to talk about how his thrillers have enhanced the reputation of the US military”.

After his talk, Clancy and I met for a drink, during which he launched more assaults on liberals, the French, Hollywood (“giving your book to Hollywood is like turning your daughter over to a pimp”), the CIA, Islamic fundamentalists, Marxist-Leninists and Bill Clinton. He told me about what he called the “Ryan doctrine” of warfare, named after the hero of his novels. “The Ryan doctrine is about taking out the principal enemy. What you do is drop four or five SAS-style guys into, say, Iraq, guys who can disappear in an instant, speak the language, have the moustaches and funny gear—you know, the hats and clothes. What they do is use ground laser designators to track and locate Saddam. Then bang, you strike from the air. The good guys win again.”

It all sounded so simple. Some, no doubt, will say that Clancy speaks with the robust common sense of the ordinary American, of which President Bush is undoubtedly one (except for the good fortune of his birth). Listening to Bush struggling to articulate the nature of his hurt, humiliation and outrage, and his stumbling attempts to offer sympathy and leadership to a traumatised nation, I have thought often of Clancy and his Ryan doctrine of warfare. “To me, the Ryan doctrine is the logical extension of military technology,” Clancy told me. “Killing people doesn’t worry me so long as you have a good enough reason. The Ryan doctrine gives you a reason.” As do irrational messianic fervour (Bin Laden) and wounded indignation (George Bush).

To Clancy, war is the “ultimate blood sport”: to deny its necessity is to deny the truth. His career as a writer may have been one long, extended patriot game, but as his country prepares to strike against a nameless and opaque enemy, he must be horrified at how adeptly Islamic terrorists appropriated the destructive impulses of American entertainment culture, making of a nation’s apocalyptic fantasies a terrifying actuality, as if they were attempting to speak to Americans in their own language. In so doing, the terrorists created instantly replicable images of catastrophe that will haunt our imaginings for ever, not least those of Tom Clancy himself, who for so long has animated our anxieties, dramatised our disasters and savoured our last moments. Except that even he could not have been prepared for what happened next—for apocalypse here and now, in New York.