I was on the terraces of the old North Bank at Highbury on that warm spring afternoon in 1989 when English football changed forever. A final year undergraduate at the time, I had travelled to London to watch Arsenal, in contention for their first championship since 1971, play Newcastle. The crowd of less than 35,000 was disappointing-the capacity of Highbury was 55,000 in those days of banked terraces, when you could wake on a Saturday morning and decide, as I had, to watch a top-flight football match. But it meant there was space to move on the terraces that afternoon, and a terrific sense of restless anticipation-even if, elsewhere in the country, a more important game was to be played: Liverpool v Nottingham Forest in the FA Cup semi-final at Hillsborough stadium in Sheffield.

I recall just before kick-off how an amplified voice, emerging from somewhere deep in the stadium, announced that the match at Hillsborough had been postponed because of a disturbance. This aroused a predictable chorus of “We hate Scousers/We hate Scousers.” Then the same amplified voice, at once more urgent and sombre, interrupted the boisterous banalities: there had been “fatalities” at the game, it said. I shall never forget the initial bewilderment and silence inside Highbury and then the whispered unease that began to sweep through the crowd as wind sweeps through a forest of trees. Arsenal won a subdued game 1-0 and we returned home to be confronted by television images of what had unfolded at Hillsborough and by newspaper photographs the next morning of the suffering of those crushed against the security fencing. Don DeLillo, in his novel Mao II, described the scenes at Hillsborough as being like something from a great religious painting, “a crowded twisted vision of a rush to death as only a master of the age could could paint it.”

What has since become memorialised simply as the Hillsborough disaster was the final act in a decade of misfortune for English football, which had included the violence by Liverpool fans that led to the death of 39 Italians when a wall collapsed at the Heysel stadium in Belgium before the 1985 European Cup final. The English game had reached a terminus, the point at which it had to modernise if it was ever to become anything more than the preserve of the white urban working-class male, a theatre of hate and of violent, often racist, excesses. The fortunes of football in this country were thus rather like those of the old Labour party: riven by factionalism, in thrall to the past, unattractive to women and urgently requiring a consensus-breaking transformation.

Today, as a result of Rupert Murdoch-inspired television money (£600m into the Premiership alone last season), English football is now at the end of the gaudiest spending spree in its history. Football is at the very centre of our style-driven, thin-spun, entertainment culture: a symbol of aggressive meritocracy, of the embourgeoisement of society, of our fascination with money and celebrity, and-in the shape of the small army of foreign players and managers-of a deracinated cosmopolitanism.

The leading English players-David Beckham, Michael Owen, Sol Campbell, Steven Gerrard-are icons beyond their sporting context, young men from working-class families whose fabulous incomes have freed them from all mundane constraint, their lives an index of thrilling possibilities, both on and off the pitch. They are, as one director at a leading club explained, “cosseted by the clubs as never before and surrounded by a team of advisers, accountants, dieticians, image consultants and lawyers, offering them constant advice. As a result, they lead lives totally removed from everyday reality.”

But what happens when you have everything you can possibly want, when all economic needs are sated before you have reached the age of 30? What does this do to a young man of little education, to his desire and motivation? What are the dangers-and the privileges? “Contemporary capitalism is prodigiously productive, but the imperative that drives it is not productivity,” writes John Gray in his new book Straw Dogs: Thoughts on Humans and Other Animals (Granta). “It is to keep boredom at bay. Where affluence is the rule the chief threat is the loss of desire.”

Tony Adams, in his memoir Addicted, offers a fascinating portrait of the life of the successful modern footballer, in both the pre- and post-Hillsborough eras. For him what motivates top footballers is not fear of failure but-as Gray says-fear of boredom. Adams is one of the most accomplished players of his generation: a former captain of England, he has been the inspirational captain of Arsenal throughout a period when the club has won ten major trophies, including this year the FA Cup and Premiership “double.” But his life, before he declared himself to be alcoholic in 1996, was characterised by isolation and intermittent depression. In Addicted, he writes candidly about the lassitude and inertia in the life of the professional footballer and of how an arrested education prevents most players from knowing what to do with their abundant leisure: the empty afternoons and long evenings away from the sound of the crowd and the camaraderie of the training ground. For Adams and others like him, drink and gambling, the twin blights of so many traditional working-class lives, offered release.

Yet, like a latter-day Jude the Obscure or Leonard Bast, Tony Adams has since remade himself through education. In Addicted, he writes of his enjoyment of poetry-of Hardy, Keats and Shakespeare-of his interest in travelling, the theatre and fine food, of his desire to learn French and of his piano lessons. Unusually for London-based footballers, who usually gravitate towards the affluent suburbs of Hertfordshire and Essex, he has bought a house in Putney to be close to the museums, galleries, restaurants and theatres of the city centre. “Today I am not just Tony Adams the footballer, I am Tony Adams the human,” he writes at the end of Addicted, a ponderous, mock confessional, new age kind of expression perhaps, but a moving admission all the same of past insularity. You close the book impressed by the courage and the peculiar grace of the man.

Adams recently announced that he would be donating £500,000 from his end-of-season testimonial against Celtic to the charity Sporting Chance, which he co-founded to support footballers with addictions. This followed Niall Quinn, of Ireland and Sunderland, who became the first footballer to donate the entire £1m earnings of his testimonial to charity. The system whereby players receive the gate receipts from a testimonial match in the form of a tax-free payment was introduced in the era of the maximum wage, when the life of the footballer was drastically constrained and often curtailed by what today would be considered routine injuries. The testimonial system in an era when the average wage in the Premiership is nearly half a million per year, excluding bonuses and endorsements, and when the top players at clubs like Manchester United and Liverpool earn between £40,000 and £80,000 per week, is a grotesque anomaly that ought to be abolished, or at least taxed. Perhaps Adams and Quinn will shame their fellow professionals into following their example.

Listening to Adams, interviewed before the cup final, you wondered if he might just be the emerging archetype of a new kind of footballer, the dedicated self-improver who uses his wealth and leisure to transcend the limitations of his background, becoming culturally middle class in a way that old socialists envisaged would happen once people were freed from hardship and toil. Or is Adams an exception in a world of stylised decadence? Is the truth of the modern game really that portrayed in the recent ITV drama Footballers’ Wives, in which the contemporary footballer was caricatured as a monolith of base behaviour and vulgar motivation?

Footballers’ Wives was a carnival of materialist excess. The characters-Chardonnay, Tanya and Jason Turner, Kyle Pascoe-and the plotlines, including attempted murder and kidnap, were lurid, cartoonish. But Footballers’ Wives sought to remind us of the self-confidence of the moneyed working class: people with no interest in becoming culturally middle class.

Liz Lake, producer and co-writer of Footballers’ Wives, was attracted to writing about football because of its popular glamour. “There’s been an absence of glamour from our screens since the days of, say, Dallas,” she told me. “We were commissioned to produce a series that was intelligent, entertaining but above all glamorous. Football offers glamour and celebrity. It’s about young men who are physical thoroughbreds and skilled at what they do, who are surrounded by beautiful girls, but who have no education and more money than they know what to do with-a combustible combination. Not all footballers are like Jason Turner [the hard-drinking, violent, promiscuous anti-hero of the series], just as all oilmen aren’t like JR Ewing. But our characters weren’t meant to be caricatures, either. We wanted the viewer to be appalled, but occasionally moved as well. The show is consistent with the context of its time. It could not have been written ten years ago.”

She is right about that, not least because ten years ago the continental sophisticate who, in the series, was a kind of Arsà¨ne Wenger-Sven Goran Eriksson compound, would not have managed her fictional football club. Liz Lake agrees that the greatest change to football in the era of the new television money was the influx of so many overseas players.

The english cup final has long been celebrated as a great national event (although it is now played in Wales). This year only seven of the starting 22 players were English. But the ethnic diversity of the Chelsea and Arsenal squads passed scarcely without comment: the transformation of the top English clubs into efficient, largely harmonious groupings of multinational, polyglot, multimillionaires has been achieved with remarkable swiftness and scant complaint. In this, football is a model of tolerant inclusiveness. The effect has been to civilise the supporters-despite sporadic hooliganism at Millwall, Cardiff and Portsmouth-to encourage a greater acceptance of difference. If a player is good enough and works hard, he is celebrated by the supporters, whether born in Senegal, as in the case of Patrick Vieira at Arsenal, or in Romford.

Gordon Taylor, chairman of the Professional Footballers Association, told me that just under half of the 896 players registered in the Premiership were now what he called “non-nationals.” He concedes that there was initial resistance from some British players to the foreign invasion, players sceptical of any change to an indigenous football culture founded on booze and blokeish bonding. Paul Gascoigne, in a notorious television outburst, accused foreign players of not understanding English passion. But in this, as in so much else, Gascoigne seems like the last maverick embodiment of an earlier footballing age, rather than the symbol of renewal that he seemed to be when he first emerged in the mid-1980s.

Of course, not all the foreigners have been a success, says Taylor. “There have been many instances of players arriving with great fanfare through the front door and then slipping away quietly through the back door, at great financial cost to their clubs. Another downside is that there are now fewer opportunities for young home players. Clubs are less willing to take a gamble on an unfinished lower division player when they can buy a more experienced international from abroad. The days when some of our best players-such as Ian Rush and Kevin Keegan-were bought from the lower division have gone. The transfer money now just circulates within the Premiership, or goes overseas.”

Taylor is impressed, however, by the sophistication of continental coaches such as Arsà¨ne Wenger at Arsenal and Gerard Houllier at Liverpool, both of whom are university graduates. These French managers have introduced new forensic training methods and approaches to diet, nutrition and exercise; they work rigorously with statistics and computers to analyse individual performances, from the number of tackles a player completes in a game to the distance he runs. They are as adept at working with the media as they are at moving between different languages. I was once present at a press conference when, without hesitation, Wenger switched from English to French and then answered a further question in German (he is, after all, the son of Strasbourg café owners).

As a student in the late 1960s, studying for an MA in English, Houllier spent a year in Liverpool while writing a thesis on inner city deprivation. He recalls standing on the Kop at Anfield, a place of romantic wonder for football fans. Houllier was enchanted to find himself amidst so much spontaneity and song: “The noise, the singing, the moving. It was swaying all the time; you could hardly see the game!” When later he returned to the city as manager of Liverpool, he was determined to reinvent the club but without violating its traditions. “Liverpool has a tradition of passing football,” he told John Williams, director of the centre for football research at Leicester University and editor of a fascinating book of essays about the game called Passing Rhythms (Berg). “It is linked to the fact that the passing is a language between people-it’s a bond between people. Everybody involved in the passing, everybody working for the same aims, and so on. Has this got something to do with tradition? It must appeal to the ‘imaginary’ of the people in Liverpool. All I know is that having been a technical director in France, it is not the same there and it couldn’t be the same. And it has got something to do with the way of living, the culture, the history of Liverpool.”

It is hard to think of an English manager speaking about the “football imaginary”; but the emergence and acceptance of Houllier, who is as much a sports scientist as he is a motivator of players, is another encouraging sign that the new cosmopolitans are winning the cultural war within the game. When he was technical director of the Institut National du Football in France, Houllier helped to establish an academy system, based around regional centres of excellence, that has resulted in the creation of a generation of exceptional, technically gifted French footballers most of whom play outside France and have an ease, range of reference and ambition that distinguishes them from the largely monoglot, monocultural British player. For too long British footballers were encouraged to subordinate individual skill to the hurtling demands of the team. English football, certainly in its late modern manifestation, was traditionally played at a ferocious, unrelenting pace; tenacity and endeavour were valued more than technique. The game was resolutely patriarchal, anti-intellectual and indifferent to continental European influence. In England, for example, we have no tradition of the libero, the gifted playmaker who invents the game from the back; superbly articulate in the language of football, he creates his own style and idiom.

Many of the most successful club managers of modern times-Don Revie at Leeds, Brian Clough at Nottingham Forest, George Graham at Arsenal-were remote, authoritarian figures who assembled squads of rugged, uncompromising British and Irish players, and then ruled them through force of personality and a culture of fear. These were teams in which the individual was subsumed into the collective. “These managers created a siege mentality among their players,” says John Williams, “and much of their bonding was done in the pub. It was very hard to integrate foreign players into such an environment. When Houllier arrived at Liverpool he wanted to blend the best of the British tradition-the aggression, the pace and intensity-with improved discipline and technique. Which meant he had to break up his largely British squad, removing disruptive elements such as Paul Ince, David James and Robbie Fowler. He believed that some of the English players weren’t serious enough about the game. They didn’t think enough about their careers and what they wanted to achieve, the kinds of people they wanted to become. It was as if they were simply enjoying an extended adolescence.”

Since the mid-1990s, English clubs have begun to introduce their own French-influenced academy systems, a consequence of which is that gifted young players are becoming not only more technically adept but also more prepared for the multiple challenges of the new game, with its hard media focus and improbable riches. “From an early age players receive the kind of good guidance that is dragging them towards a greater responsibility,” says John Williams.

Yet, poignantly, many of the new footballers are prisoners of their wealth and celebrity, liberated only into a life of seclusion, sterility and repetitive routine. “They buy their flash houses with their high walls and spend most of their time in them, playing their computer games or their expensive DVD systems,” says Ian Ridley, the football writer and co-author of Addicted. “The demands of the modern game are so great that players can no longer live as they once did. If they attempted to go down the pub or travel on public transport, they would be thumped.”

But will the riches in the new game draw in footballers from a wider social background, as is the case in some European countries? “Footballers here have always been drawn from a narrow section of society, and that is unlikely to change,” says Ridley. “I used to coach a local team in Hertfordshire. They were a nice bunch from middle-class homes but I knew that whenever they played a team from a rough area they would lose. Why? It was because the culture of football is ingrained in these people’s lives. They are shrewder, more streetwise, they seem to understand the game better, its demands and mentality.”

Which means that, if Ridley is right, little Brooklyn Beckham will never become, like his father, a professional footballer. The England captain has spoken again and again of how he wants his son to have a “normal life” and be able to attend the school of his choice free from interference. Beckham imagines waiting for his son at the school gates, as his parents once waited for him at his comprehensive in the east London suburbs. That will not happen; Brooklyn Beckham is destined, like so many children of the new football plutocrats, to attend an elite fee-paying school, where he will receive the kind of education that prepares him for anything but a life in football.

Of present Premiership professionals it is thought that only one, Matt Jansen of Blackburn, attended a public school. But when I spoke to a former player, Alan Smith, who now writes for the Daily Telegraph, about players’ attitudes to education he said that all but one of those with whom he used to play at Arsenal send their children to fee paying schools, as does Smith himself. Smith is something of an oddity in the English game: grammar school-educated, he was in his first year at university when he joined Leicester City. “I’m not sure that we are going to see a new, more culturally rounded footballer,” he says. “Most players are from working-class backgrounds and from the ages of 12 to 16 all they want to do is play football. But I did both: I played football and concentrated on my education.”

Does he think that more players will donate substantial sums to charity? “Players like Adams and Quinn are very much the exception. When I first knew Tony, he was a typical Essex lad who liked a drink. Since then he has altered beyond all recognition, but that’s as much to do with his own personal difficulties as it is to do with changes within the game itself. When I was with Niall Quinn at Arsenal, he always had socialist principles and did a lot of work with the charity Shelter. That’s the way he is.”

When Alan Smith retired in 1995, all the Arsenal players were “earning roughly the same.” Within three years, Smith’s former teammates, such as Ray Parlour, Lee Dixon, Steve Bould and Adams, had seen their “salaries quadruple” to about £1m per year. Smith has no regrets about missing out on the hyper-wages-“I had a great career, and I won things,” he says. “In any event, I was very well paid and managed to put something aside. But I still have to work.”

The young players of today at clubs such as Arsenal and Manchester United, so long as they remain free from injury, will not have to work when they retire from playing football. But what kind of people will they be? How do they envisage their roles in wider society? For whom will they vote (when the England World Cup squad of 1982 were asked their political preferences only one player, Paul Mariner, said that he voted Labour)? What, if any, will be their cultural aspirations?

Perhaps the future belongs neither to the Tony Adams self-improvers nor the unreconstructed Jason Turners-or his real life equivalents like the Leeds pair of Lee Bowyer and Jonathan Woodgate known for their boozing and brawling. Perhaps, rather, the future is represented by players like the robotic Michael Owen, players who seem as bland and stilted off the pitch as they are creative on it. Owen comes from a football family, and from early adolescence he has been groomed for life as a professional. In everything he does, the way he speaks and behaves, he is the definitive new footballer as product, a footballer who has, like Alan Shearer and many others, all the spontaneity of canned laughter, so controlled is the environment in which he moves.

The past decade has been the belle époque of the professional footballer. From 1992 to 2000 the combined wages of all Premiership players rose from £54m per season to £319m, according to Deloitte & Touche. Meanwhile, the average season ticket has risen in price by nearly 30 per cent in five years. The world’s best players are earning as much as £100,000 per week, and many millions more through endorsements and sponsorship. At the same time, ten of the 20 clubs in the Premiership and 90 per cent of the 72 clubs in the Football League are now operating at a loss. And 600 professionals from the lower leagues in England and Scotland were expecting the sack this month, partly because of the collapse of ITV Digital and the projected fall in television income.

Despite the shake-out, football will remain rich and glamorous-at least at the top. But many fans think that something is missing, there is nostalgia for the certainties of the old game with its rugged tribalism. If you read the new football literature and the fanzines of the most committed supporters you will find expressed the belief that the game has lost its soul: the football may be more stylish but somewhere in the rush to embrace modernity much of the charm has disappeared. More than in the past, money buys success and the established metropolitan clubs can insulate themselves against challenges from below. It is no surprise, perhaps, that only 6.3m people watched this year’s FA Cup final on television, the lowest ever figure and fewer than watched the final of the world snooker championship the following day.

Yet the persisting appeal of football is that, in the end, it provides a dramatic narrative, a consistency of interest through time. It is, in our secular, post-ideological age, the closest thing we have to a national conversation. There’s something mystical about fandom that defies rational explanation. Supporting a football team is about faith. Once you have chosen your team, which usually happens in early childhood, that’s it: you have crossed a border that, for true fans, is irreversible. The players, managers and even the stadium may change, the chairmen and owners come and go, but something endures about a football club, something akin to the Aristotelian idea of metaphysical substance behind all mutation. When Houllier speaks of the “football imaginary,” of a romantic attachment between Liverpool football club and the city, its history and struggles, he exhibits an empathy with the fans that many of football’s modern mercenaries, moving from city to city and country to country, no longer understand.

I was at the Millennium Stadium in Cardiff when Arsenal lost 2-1 to Liverpool in the 2001 FA Cup final. The London team controlled the game but conceded two goals from Michael Owen in the last eight minutes. It was a melancholy end to an entertaining game; but I was not too disappointed-because I stayed on to watch as Houllier and his young team thrilled to the Liverpool supporters as they swayed and sang “You’ll Never Walk Alone,” their famous anthem of loss and reconciliation. Here against the brilliant blue of the sky of a May afternoon was a kind of secular worship with its own eschatology, not exactly redemption for what happened at Hillsborough, but a testament all the same to the enduring wonder of the game, no matter how debased by commerce-a football imaginary indeed.