Cameron made his comments minutes before Andy Coulson, his former Downing Street communications chief, was arrested on Friday on suspicion of phone hacking and corruption.

The prime minister refused to bow to pressure to apologise for his decision to give Coulson, former editor of the News of the World, a "second chance" after he was forced to quit the Sunday tabloid over the phone-hacking scandal in 2007.

Cameron said the shocking new allegations around practices deployed by the News of the World "and possibly elsewhere" was a "wake-up call" for the country as he unveiled a "comprehensive plan to put it right".

"The truth is, we have all been in this together – the press, politicians and leaders of all parties – and yes, that includes me."

He announced a package of measures, including a judge-led public inquiry; a second inquiry into press ethics, due to begin this summer; and said the Press Complaints Commission (PCC) should be replaced with a fully independent body.

But he came under pressure to comment on individuals in senior positions at the time including Coulson, Rebekah Brooks, now chief executive of News International, and James Murdoch.

Murdoch made a statement on Thursday which included the admission of "repeated wrongdoing" at the paper and the revelation that he had personally approved out-of court payments to cover up illegal phone hacking. When asked about that admission, Cameron told journalists that everyone who had questions to answer should be approached by police.

"The police have got the resources and the skills to follow the evidence wherever it leads. To question anyone no matter how high or low," said Cameron.

"The statement yesterday leaves all sorts of questions to be answered. The police must feel they can go wherever they need to question anyone."

Cameron refused to bow to calls to apologise for hiring Coulson and bringing him into the "centre of the government machine".

The prime minister remained unwavering about his decision to give the Coulson a "second chance" by appointing him director of communications. Coulson resigned in January but remained a friend, Cameron said.

"The decision to hire him was mine – and mine alone – and I take full responsibility for it," he said.

Cameron insisted he commissioned a firm to carry out a background check before employing Coulson.

"I took a conscious choice to give someone who had screwed up a second chance," the prime minister said. "He worked for me, he worked for me well, but actually he decided in the end the second chance wouldn't work; he had to resign all over again for the first offence."

Alan Rusbridger, editor of the Guardian, revealed on BBC2's Newsnight on Thursday that he had sent a message to Cameron via a senior aide to warn him before last year's general election that taking Coulson to Downing Street could be fraught with difficulties further down the line.

Asked what inquiries he had made before employing Coulson, Cameron said: "No one gave me any specific information. Obviously I sought assurances, I received assurances. I commissioned a company to do a basic background check, but I'm not hiding from the decision I made.

"I made the decision, there had been a police investigation, someone had been sent to prison, this editor had resigned, he said he didn't know what was happening on his watch but he resigned when he found out, and I thought it was right to give that individual a second chance."

The prime minister said he and Coulson spoke before Christmas about a parting of the ways.

"It wasn't in the light of any specific thing, it was a sense that the second chance wasn't working," said Cameron.

"He had been given a second chance, he was doing, I thought, a very good job working very hard for the government of the country, but was finding it very difficult to do his job because of all the swirling accusations about the News of the World.

"And so the conclusion he came to, I think rightly, was 'the second chance is not working, I have got to resign all over again for what happened back then.'"

Cameron said they discussed the hacking allegations while Coulson was employed but he never had reason to doubt "the assurances he had given me and I accepted".

On their contact since Coulson's January resignation, he said: "I have spoken to him, I have seen him, not recently and not frequently, but when you work with someone for four years as I did, and you work closely, you do build a friendship and I became friends with him. He became a friend and is a friend."

Questioned on who may have lied and about what, Cameron said: "I don't know what these people at News International did know or didn't know. Frankly, I don't think any of us know what they did know or didn't know.

"The key thing is they are going to be investigated by the police and when they get investigated by the police and when the truth is out, it won't be a question of whether or not they have jobs or whether or not they resigned from those jobs, it's a question of whether they are going to be prosecuted, whether they are going to be convicted, whether they are going to be punished."

Cameron said the scandal was not about journalists on one newspaper, but also involved the police and politics. "You can downplay it and deny the problem is deep – or you can accept the seriousness of the situation and deal with it," he said. "I want to deal with it."

Cameron, who was previously reported to be resisting a full public inquiry led by a judge, said he and his deputy prime minister, Nick Clegg, had agreed that a judge should preside to ensure the inquiry was "totally independent" in the search to get to the bottom of the allegations. "No stone would be left unturned, he said.

While the inquiry could not start officially until the police investigation had concluded, the prime minister said some elements would begin now, such as consulting on its terms of reference and remit.

Cameron, who has been pressed by Miliband to show leadership without "fear or favour", said a second inquiry would begin "at the earliest available opportunity" into the wider lessons on the future of the press, and would be conducted by a panel of "credible figures" from a range of backgrounds.

Cameron said the PCC had "failed" and shown itself to "lack rigour". The inquiry would recommend what an entirely new system of regulation should look like.

"But my starting presumption is that it should be truly independent ... independent of the press, so the public will know that newspapers will never again be solely responsible for policing themselves," he said.

Questions had been raised not just about the media and the police but politicians, too, he added. Citing a number of warnings coming from the media, select committees and the information commissioner over recent years, Cameron said the government of the day – and the Tories in opposition had failed to respond, partly because any move by politicians to make the case for tighter regulation of the media risked being portrayed as parliament trying to stifle a free press or even free speech.

But he also admitted that the less "noble" reason was that party leaders of all political hues were so keen to curry favour with media editors and proprietors that they turned a "blind eye" to the need to get on top of bad practices. Cameron, who has been linked to private family dinners with Brooks, conceded that Labour and Tory leaders alike had spent time "courting support rather than confronting problems".

Likening the scandal that reached its tipping point this week to the MPs' expenses scandal in 2009, Cameron said: "The people in power knew things weren't right. But they didn't do enough quickly enough – until the full mess of the situation was revealed."

Now that the "music has stopped", it was time to change, he said.

Sign up for the Guardian Today

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.