Ed Miliband’s defeat and resignation are a personal humiliation and a family tragedy.

He challenged his elder brother David for the leadership of the Labour Party in 2010, effectively destroying their relationship, because he was convinced it was his destiny to become prime minister and to lead Britain in a bright new direction.

In his dignified resignation speech, Miliband said he’d lost the election but ‘not the argument’.

This was the statement of a deluded man. He lost the election and the argument. If Labour doesn’t understand this, and adapt accordingly, it is in deeper trouble than even I believed.

Driven by a kind of messianic self-belief, Miliband was Labour’s most unashamedly Left-wing leader since Michael Foot, whose 1983 election defeat condemned the party to a long, painful period in the wilderness as Margaret Thatcher accelerated her transformation of Britain.

On the occasions when we met, he told me again and again that the financial crisis and the consequent Great Recession had created what he called a moment of great opportunity for the Left.

Perhaps they did in Scotland, but certainly not in the seats in the Midlands, Home Counties and southern England that Labour must win if it is ever to return to power.

‘Under Miliband, we had nothing to say to the faraway towns of England,’ one senior Labour figure told me.

By which he meant, Miliband’s cerebral socialism might have been popular among metropolitan liberals but it emphatically did not resonate with the skilled working and lower middle classes in small towns in places such as Essex, Bedfordshire, Kent, Hertfordshire …

As I wrote last November, in a column that precipitated a leadership crisis (there were, for a time, reports of attempted coups and secret letters circulating), Ed Miliband has a quasi-Marxist worldview.

His great obsession is political economy. He believes that if you change the economic base of a nation, you change the soul of its people. Rent controls, energy price freezes and heavy redistributive taxation were the means by which he believed the State would set about taming the ‘predatory’ forces of capitalism and building the ‘good society’.

As soon as he became leader, Miliband was eager to distance himself from the Blair years, even the successes, and this merely angered many of his MPs.

His focus on economic matters was exceptionally narrow. He had a theoretical understanding of business and poverty but no practical experience of either.

Unlike Blair, who understood the value of middle-class aspiration, Miliband seldom spoke about education (even though state academies were the creation of New Labour) or Britain’s place in the world. He never discussed public service reform or wealth creation.

Far too late, he made a desperate attempt to reframe Labour as the party of fiscal rectitude by including a so-called ‘budget responsibility lock’ (which guaranteed that every policy in Labour’s manifesto was fully funded without requiring any additional borrowing) — a matter of months after forgetting to mention the deficit in his party conference speech.

‘Ed spent his whole leadership in the bunker,’ I was told this week by one despairing former confidante. ‘Don’t forget, he spent all those years in the bunker alongside Gordon Brown and Ed Balls. That shaped him. He had a siege mentality, and he thrived on it.’

Meanwhile, Miliband failed to glimpse, even out of the corner of his eye, what was unfolding in Scotland, where Labour used to weigh the votes in constituencies that, in some cases, had all the characteristics of rotten boroughs.

The party sent its brightest talents to London and introduced a botched devolution settlement. George Robertson, the former Labour defence secretary, boasted that devolution would ‘kill Scottish nationalism stone dead’.

Instead, it empowered the SNP and fired its ambitions, allowing Alex Salmond and Nicola Sturgeon to claim all Scotland’s successes as their own, while its failures could be blamed on Westminster.

The Labour collapse in Scotland has been dramatic in its totality and rapidity, but in truth it was a long time in the making.

A few prescient politicians such as Douglas Alexander, the former shadow foreign secretary who lost his Paisley seat to a 20-year-old student, understood how nationalism and identity politics were profoundly changing his homeland.

But Alexander, who led David Miliband’s doomed leadership campaign, was another senior colleague not fully trusted by Ed Miliband.

To the last, Miliband remained complacent about the rise of the SNP and ended up even more unpopular in Scotland than David Cameron. Not once during the election campaign did he feel confident enough to take his message directly to the Scottish people, on the streets or at public meetings.

His visits there were few, and always tightly choreographed. It was almost as if he were some kind of fugitive, being hurried in and out of a hostile land.

Labour MPs often complained to me about Miliband’s small team of closest advisers and the effect they were having on him. Having won the leadership because of the overwhelming support of the unions, he never really trusted many of his MPs.

In the early years of his leadership he felt isolated, and was especially suspicious of Ed Balls, with whom he worked as an aide under Gordon Brown at the Treasury and whose respect he never fully won.

Miliband surrounded himself with a small group of like-minded, scholarly, middle-aged men: most notable were Stewart Wood, a former Oxford academic, and Marc Stears, also an academic and an old friend of Miliband’s from Oxford University who wrote many of his speeches.

It created a peculiar kind of group-think. Their meetings had the atmosphere of lofty seminars; but then no one around Miliband had run or set up a business.

Throughout his leadership and encouraged by his loyal aides, Miliband liked to style himself as an ‘insurgent’, challenging a nefarious establishment: big business, the multinational banks, the Press barons, the energy companies.

Some of his positioning was bold and, indeed, courageous — especially when he defied David Cameron and refused to support British strikes against the Assad regime in Syria in 2013.

But it never seemed to dawn on him that he, too, was an emblematically establishment figure. He just happened to be a member of a different establishment from the Prime Minister. That of the liberal metropolitan elite.

In my view, a more nimble leader than Miliband, one with broader life experience, would have seized the opportunity presented by the collapse of support for the Lib Dems and the rise of Ukip, which divided the Right and caused deep unrest among the Tories.

A more nimble leader would have looked beyond Labour’s core vote and attempted to appeal not just to a faction but to most of the people, most of the time.

The British are, on the whole, fair-minded and generous spirited. They are rightly sceptical of ideology and grand schemes to remake society. And, despite the blindness of the polls, it’s clear that they made up their mind about Ed Miliband almost as soon as he became Labour leader.

As it turned out, nothing that he did or said during the protracted election campaign — no boast about his toughness nor late-night tryst with Russell Brand — would change the people’s fundamental view that he was simply not up to the job of being prime minister.

Now Labour has to clear a path through the rubble.