Aid agencies have reported numerous human rights abuses in Somalia, where the UN and US have failed to restore stability.
The country's president, prime minister and parliament, appointed last year as part of a regionally-sponsored peace process, are still holed up in the Kenyan capital, due to insecurity in Mogadishu.
"Greater efforts should be made to enforce the arms embargo in Somalia," said Annan's report to the Security Council, made public on Tuesday.
"Reports indicate large-scale violations of the arms embargo, not only by extremist groups and militias, but also some members of parliament," the report said.
"The importation of explosives and heavy weapons is especially worrisome.
"Small arms proliferation is a major concern that needs to be addressed in the longer term, but the presence of large quantities of heavy weapons like tanks, artillery, anti-aircraft guns, multi-barrelled rocket launchers and heavy mortars poses a more immediate problem," Annan wrote.
Somalia, the only country in the world without a functioning government, plunged into chaos after Muhammad Siad Barri was toppled in 1991.