In early February 2000, on assignment for Rolling Stone magazine, David Foster Wallace spent a week on the campaign trail with John McCain, the no-nonsense, straight-talking, anti-Washington candidate in the race for the Republican presidential nomination eventually won by George W. Bush. Wallace, who was 37 and considered to be the most innovative literary novelist of his generation, took on the commission because there was something about the Vietnam hero-turned-politician that fascinated him – his backstory, his candour. Could he be, as Wallace put it, “for real”?

Wallace’s Rolling Stone article (later expanded and republished as a short book) is one of his best and most emblematic works because it can be read as a hilarious take on the McCain 2000 primary campaign but also as a philosophical investigation into the question of personal identity. Wallace was especially interested in the years McCain spent in solitary confinement as a prisoner of war after his fighter plane was shot down over Hanoi and he had to eject. For Wallace, McCain’s cell, his “dark box”, becomes a metaphor for human isolation and for the radical separateness of consciousness.

Wallace was tormented by one defining question: was the world anything more than a tissue of representations? A hard philosophical sceptic, he felt imprisoned inside his own head, his dark box. “There is this existential loneliness in the real world,” he told Laura Miller, the co-founder of Salon.com, in an interview collected in Conversations with David Foster Wallace. “I don’t know what you’re thinking or what it’s like inside you and you don’t know what it’s like inside me.” It was his belief (more a hope, as it turned out) that “in fiction I think we can leap over that wall itself in a certain way”.

But in his life Wallace, who suffered from clinical depression, kept slamming into that wall of separation between the self and the world. He longed to make connections – with other people, with other minds. He longed to understand better, to be free from the tumult and the pain that he felt, every day, without respite.

And yet, for all his suffering, Wallace was a wonderfully exuberant comic writer and ironist. His 1,000-plus-page novel Infinite Jest (1996) – the book that made him famous and inspired a younger generation of writers, led by Dave Eggers and Zadie Smith, to be bolder, more experimental in how they went about their business – is often astoundingly funny. It is also piercingly sad.

The novel is set in an indeterminate near future. The US, Canada and Mexico have merged to form the Organisation of North American Nations (ONAN – get it?), in which the influence of television, multinational corporations and advertising has become even more sinister and all-pervasive. Much of the action takes place in the hermetic worlds of a tennis academy and a halfway house for recovering addicts. A third thread concerns the antics of a group of Québécois separatists, many of whom travel around in wheelchairs (are you still with me?). Everyone seems to be searching for the master copy of a film named Infinite Jest. It was made by the father of a disturbed tennis prodigy we meet at the start of the book, Hal Incandenza (Wallace’s fondness for zany character names, his low-level paranoia and ironic extremism reveal a debt to Thomas Pynchon and Don DeLillo). The secret of the film Infinite Jest is that it’s so ecstatically enjoyable that it renders immobile, even kills, anyone who watches it.

The novel has more than 100 pages of footnotes and does not so much end as leave the reader suspended in mid-air, with nothing resolved. (In the jargon of post-structuralism, with which Wallace was familiar, it resists closure.) It would be a mistake to read it autobiographically even if, as DT Max illustrates in his scrupulous and affecting biography, Every Love Story is a Ghost Story, much of Wallace’s own story is encoded into it.

Born in 1962, Wallace grew up in central Illinois. His father was an academic philosopher and his mother, a fastidious grammarian, taught English at a local community college (he inherited from her a mania for correct usage). Wallace excelled at sports at school and was good enough at tennis to have made it as a pro, except that in his mid-teens he started to smoke too much marijuana and stopped training hard. (Curiously, Max makes no mention of the link between heavy marijuana use and mental illness.)

Wallace never lost his flair for and interest in tennis. One of his finest long essays is a meditation on the beauty and brilliance of Roger Federer. But like Hal, whom we first meet as he is having a breakdown, Wallace turned out to be much more than a sportsman. “I could, if you’d let me, talk and talk,” says Hal. “Let’s talk about anything.” And off he goes, riffing on Kierkegaard, Camus, Hegel and Hobbes. “I could interface you guys right under the table,” he tells his interrogators at the tennis academy. “I’m not just a creatus, manufactured, conditioned, bred for a function.”

This is the authentic voice of so much of Wallace’s writing: cerebral, vital, hyper-articulate, ever alert to the defining particulars of the age, moving between high and low culture, wised-up. In conversation, he could be tyrannical, overwhelming, eager to demonstrate just how smart he was, the smartest guy in the room. Once asked how he knew so much, he said: “I did the reading.” He did. He could be extremely funny. He was a parodist and mimic who could do all the voices. He spent many hours of most days watching television, of varying quality. He was interested in everything, except golf. And, like Hal, he could talk about anything.

As Max reveals, Wallace collected pathologies. He was addicted to alcohol (he was forced to become teetotal), marijuana, tobacco (which he smoked, chewed and spat), television, even sex (before his eventual marriage to the artist Karen Green in 2004, he moved in and out of relationships, ceaselessly on the lookout for the next woman, the next casual encounter). He once wondered aloud to his friend Jonathan Franzen whether his purpose in life was “to put my penis in as many vaginas as possible”. “Sex,” says Max, “filled a place in Wallace that nothing else could.”

At Amherst, where Wallace wrote a philosophy thesis and completed a draft of what became his first novel, The Broom of the System, he was considered the outstanding student in his year. He also had several breakdowns, and so sick was he with suicidal depression that he had to return home for a long period. Later, following another mental collapse, he was sectioned and put on suicide watch while a graduate student in philosophy at Harvard. For the rest of his life his stability was dependent on the antidepressant Nardil.

Like many American writers, Wallace became institutionalised and seemed unable to cope outside the supporting structures of the university system, with its generous stipends, grants and assistant professorships. He studied for a creative writing MFA in Arizona and then taught at various universities. He liked his hair long, like a late-1970s Argentine footballer, and wore a bandanna, which became his signature sartorial statement, like Wittgenstein’s open-neck shirt or Tom Wolfe’s white suit.

As his fame grew, Wallace’s friendships with other writers deepened, notably with Franzen, with whom he maintained a tense rivalry, the memoirist Mary Karr, with whom he had a relationship, and DeLillo, whose worldly calm and prodigious output he envied. Max quotes extensively from his letters to Franzen and DeLillo. These are questing in tone, rigorously intelligent and painfully revealing of his torment – his longing for fame, his anguish at his failure to finish another novel after Infinite Jest, his search for a happiness that’s unavailable.

Wallace wasn’t content with being merely recognised as accomplished and influential. He wanted to be the best of his generation. More than that, he wanted to be a great, canonical writer. He wanted to write novels that “will be read 100 years from now”, he told a close friend from Amherst, as he was starting out.

In 2007 Wallace became increasingly agitated about what long-term dependency on prescription drugs was doing to his health. He was struggling with a novel on which he had been labouring for years and wanted to discover if he could write better if he came off Nardil. “He was a perfectionist,” Franzen told Rolling Stone, for a piece collected in Conversations. “He wanted to be perfect, and taking Nardil was not perfect.”

Wallace never got better. In September 2008, on a luminous Californian afternoon and with his wife out of the house, he hanged himself, but not before writing a two-page note to Green and arranging the pages of the manuscript of his unfinished novel, which he left in the garage where he knew it would be found. (It was published as The Pale King in 2011.)

In life, because of his literary glamour, the outsider image he cultivated and the cerebral dazzle of his work, Wallace attracted groupies who clustered and swooned at readings. In the years since his death, the cult of personality around him has merely intensified, which often happens when an artist of influence commits suicide or dies young or unfulfilled, to the irritation of Bret Easton Ellis for one. Last week, while reading Max’s book, Ellis began tweeting of his contempt for Wallace and those who revere him: “Reading DT Max’s bio I continue to find David Foster Wallace the most tedious, overrated, tortured, pretentious writer of my generation ... “; “Saint David Foster Wallace: a generation trying to read him feels smart about themselves which is part of the whole bullshit package. Fools.”

A tragedian as well as comedian, Wallace was, as Hamlet says of Yorick, a “fellow of infinite jest, of most excellent fancy”, but Ellis, whose numbed, affectless style Wallace once spoofed expertly in a short story, in his hostility takes no account of how confused Wallace could be. He was an intellectual exhibitionist, for sure, but he was also profoundly lost. To read him is to encounter a writer of boundless imaginative gifts, a moralist as well as a maximalist whose mission was to represent in fiction how it felt to be alive, here and now, in a television-mediated, information age; to capture, as Max puts it, “the everything of America”. The form of the conventional realist novel was insufficient to the ambition of his self-appointed task, which is why his novels and stories are also an experiment in forms, hence all those different registers, clarifying footnotes, parenthetical interruptions, metafictional tricks and looping digressions. He was bursting with so much to say – and restless in his desire to find new ways of saying it.

Little escaped Wallace’s attention, nothing was beneath his notice. He wrote with the same curiosity and manic attention about porn, rap and tennis as he did about mathematics, philosophy and literature. His work is saturated in detail; the reader experiences information overload.

In remarks from a Wallace memorial service in New York in October 2008, published in a new book, The Legacy of David Foster Wallace, the American writer George Saunders said his friend’s prose had a “special variety of openness that I might call terrified-tenderness: a sudden new awareness of what a fix we’re in on earth, stuck in these bodies, with these minds”.

Wallace did feel encaged in the self. He used to say that he felt at times as if his head would explode. It could never be enough for him to know that his friends – Franzen said that he had a “beautiful, questing innocence” – and many readers loved him and that though he felt isolated, he was not alone. And yet, he did reach out beyond his loneliness, he did somehow make a connection – through his work, which will continue to be read long into the future as each new generation discovers the sad, extraordinary man who was David Foster Wallace.