The ministry apologised to Musa's family and the other men, opening the way for potentially large compensation claims.
In a statement to parliament on Wednesday, Des Browne, Britain's defence secretary, said: "After wide consultation ... I have decided that the right thing to do is to establish a public inquiry.
"Overall, the conduct of tens of thousands of our people in Iraq has been exemplary; it is a tiny number who have caused a stain on the reputation of the British Army."
A judge will be appointed to lead the inquiry and its scope will be announced later. The inquiry report will be published.
The government had resisted calls from Musa's lawyers for such an inquiry, arguing that a court martial over his death last year, which found one soldier guilty of unlawful conduct, was sufficient.
But pressure for a public inquiry grew when Britain's highest court, the House of Lords, ruled that European human rights law did apply to British troops serving in Iraq.
'Nothing to hide'
General Richard Dannatt, Britain's army chief, said the army needed to be sure that there was "nothing systemic" in the British army that led to the abuse.
"We have looked very thoroughly at ourselves already. We have, we believe, put right those things that need to be put right. I'm not conducting a witch hunt here and I'm not lifting stones that I'm fearful of what I will find underneath."
He said the army had "nothing to hide".
Phil Shiner, a lawyer representing Musa's family, said the inquiry was the "start of a long process of accountability".
Difficult questions had to be asked about why interrogation techniques banned by Britain in 1972, such as hooding, had been used in Iraq, he said.
"There were many other incidents which are not going to be picked up in this inquiry. What we really need is a single inquiry into the UK's detention policy in southeastern Iraq," he said.
Musa, whose wife had died two months earlier of a brain tumour, was 26 when he died and left behind two children.
While the image of the US military in Iraq was tarnished by abuse at Abu Ghraib, Britain's smaller forces have largely escaped public censure, though 21 soldiers have been court-martialled for abuses in Iraq.
Seven officers and soldiers were court-martialled in the case of Musa and the others, but only one was found guilty after admitting mistreatment of prisoners.