David Miliband today confirmed he was quitting frontline politics, but promised to serve the Labour party and his younger brother's leadership to his utmost from the backbenches.

Ed Miliband, the Labour leader, responded by saying his door was always open for his brother to return to the front bench in the future.

The defeated Labour leadership candidate insisted that stepping back from the political limelight was the "cleanest and clearest thing to do although not the most obvious or easy thing to do" in order to allow Ed Miliband to set the future direction for the party free of permanent scrutiny of any potential differences between the two men.

Ed Miliband, looking notably relaxed as he spoke outside Labour's Manchester conference, said that while it would have been "fantastic" to have had his older brother as part of his team, David's decision was "thoughtful and gracious".

"My door is always open for him to serve in the future," Ed Miliband said.

Other colleagues paid tribute to David Miliband, who has served on the frontbenches since 2002 – a year after he was elected to parliament – and was seen as a frontrunner in the Labour leadership until the final few days of the race, when the odds switched in favour of his younger brother.

The shadow foreign secretary and MP for South Shields, who was defeated in the leadership race by his younger brother by a narrow margin on Saturday, confirmed he had decided not to put his name forward for the shadow cabinet elections in a letter sent to his constituency party chair, Alan Donnelly, this afternoon.

The party was his younger brother's to lead and he needed to be able to do so as "free as possible from distraction", he wrote.

In a round of interviews broadcast after the 5pm deadline for shadow cabinet nominations passed without his name on the list, David Miliband said the "easy thing" would have been to have run for the shadow cabinet, but it would have been a route to "real difficulty" because of constant scrutiny of body language and comments he made in an attempt to detect policy fissures between the two brothers.

"I decided I can best serve the Labour party and the leadership from the backbenches in the House of Commons, rather than the shadow cabinet," he told Sky. "I think that will give Ed the space to lead the party as he sees fit without any distractions or suggestions of splits and divisions and it also gives me the chance my own batteries and think the big issues I want to make a contribution to in the future."

In a separate interview with the BBC, he cited the way the media picked up on his exchange with deputy leader Harriet Harman when she applauded Ed Miliband's declaration that the Iraq war was wrong, showed how difficult it would have been if he had remained.

"I think that raising a wry eyebrow with Harriet yesterday shows the dangers that can come," he said.

"I think that it is really important that no one is able to use anything, the merest sneeze or body language or comment, to divert from the really important task that the party has, that Ed has, and that I want to support him in doing."

Miliband insisted that the differences between his brother and himself on policy "are very small indeed". Asked about Iraq, Miliband insisted that the two brothers shared a very strong set of shared values and vision embodied in the Labour party for the future of the country.

"I'm not going to be tempted in any way at all down the track of differences and divisions. We are a united Labour party," he said.

Tributes poured in as Miliband bowed out from the front line. Lord Prescott, the former deputy prime minister, said that the party owed him "an enormous debt of gratitude".

"He helped create a record that we can be proud of and I respect the fact that during the leadership campaign he chose to defend it," he wrote on his blog.

Alan Johnson, the shadow home secretary, who backed David Miliband's leadership bid, was one of several colleagues to declare his decision as the right one in the circumstances.

Johnson said: "To remain in the shadow cabinet would invite constant scrutiny of their relationship and endless discussion of every nuance, however trivial. This decision will make it easier for Ed to flourish as leader whilst allowing David to pursue other political interests as well as working tirelessly for his South Shields constituency."

Labour frontbencher Tessa Jowell said: "He has done the right thing for himself, and for his family, and he has definitely done the right thing for the party. This has now given his brother Ed the space to run the party."

Miliband said he was convinced he had made the correct decision to stand down from the shadow cabinet.

"I am absolutely certain that for the party and the country, which has driven me in every political decision I have taken, this is the right thing to do."

He stressed that he was not ruling out a future return to the Labour frontbench if his brother asked him to serve in the future.

"I will always make the decision that I think is right for the party and for the country and I think I can't do better than that," he said.

He said he was proud of his leadership campaign, despite losing by a "narrow margin" to his brother.

Miliband admitted that the campaign had been "tough, tougher when your brother is running", but it had set a tone for the party's future direction.

Asked whether the two brothers had really thought through the campaign and its aftermath, Miliband declined to address the question directly but admitted the situation had been "unusual".

He said he was glad the contest had not become a "bloodbath", and said "good humour and good cheer were kept up right the way through and good family relations as well".

"I got more of a sense last year that Ed wanted to run, and he was perfectly within his rights to do so, so it was not a complete shock," he said.

"Although once the campaign gets going it is then a long and arduous process, a gruelling process. But it didn't become the bloodbath that many people predicted."

Pressed on this point, he said: "I am not dead - I am still here."

The Conservatives said that the decision of David Miliband, often characterised as a Blairite, not to serve on the new leader's front bench "speaks volumes" about the direction his brother was taking the party.

Lady Warsi, the Tory chairman, who has sought to frame Ed Miliband's victory as a sign that Labour is veering to the left, said: "David Miliband was a leading architect of New Labour. The fact that he doesn't want a place in Ed Miliband's shadow cabinet speaks volumes about the direction in which the new leader is taking Labour."

In his letter, Miliband said he feared "perpetual, distracting and destructive attempts to find division where there is none, and splits where they don't exist, all to the detriment of the party's cause", if he was elected to the shadow cabinet.

Miliband plans to use his spare time to spend more time with his wife, Louise, and two sons, Isaac and Jacob, and to further explore themes from education to the environment and to foreign policy.

He wrote: "The party needs a fresh start from its new leader, and I think that is more likely to be achieved if I make a fresh start. This has not been an easy decision, but, having thought it through and discussed it with family and friends, I am absolutely confident it is the right decision for Ed, for the party, and for me and the family."

Miliband went on: "Any new leader needs time and space to set his or her own direction, priorities and policies. I believe this will be harder if there is constant comparison with my comments and position as a member of the shadow cabinet. This is because of the simple fact that Ed is my brother, who has just defeated me for the leadership."

His announcement overshadowed the publication of the list of 49 MPs who have put themselves forward for election to Ed Miliband's first shadow cabinet.

Labour MPs will elect 19 of them, and Miliband will choose which jobs they fill. Rosie Winterton was the only person to put herself forward for the separate election of chief whip, and will replace Nick Brown, one of Gordon Brown's close allies.