Thursday’s claim by Clare Short drew an angry response from the United Nations.
Blair declined to address the claim, beyond saying British security services acted within domestic and international law.
But the UN declared any such operation would be illegal.
"We want this action to stop if indeed it has been carried out," UN chief spokesman Fred Eckhard told reporters.
"It undermines the secretary-general's conduct of business with other leaders. It is therefore not good for the United Nations' work and it is illegal."
Britain's UN ambassador talked to Annan on Blair's behalf.
Short's allegation comes a day after Britain dropped charges against a translator who admitted leaking a top-secret US document seeking London's help in bugging United Nations members in the run-up to the war.
The former aid minister, who resigned after the war but was in government during the period when London and Washington sought UN authorisation for military action, said Secretary-General Annan's office had been specifically targeted.
'Spying for some time'
"In the case of Kofi's office, it was being done for some time," Short told BBC Radio. "I read some of the transcripts of the accounts of his conversations."
Blair reacted heatedly to his now frequent critic, saying Short was undermining the intelligence services and British security.
UK Prime Minister Tony Blair is
embroiled in another scandal
"The fact that those allegations were made...is deeply irresponsible," he told a news conference from his Downing Street home.
"We are going to be in a very dangerous situation as a country if people feel they can simply spill out secrets or details of security operations, whether false or true."
Iraq has become a political nightmare for Blair. Ten months after Saddam Hussein was toppled, none of the banned weapons Blair claimed Iraq had primed for use has been found.
The premier's public trust ratings have slumped and many in his Labour Party feel betrayed to the point of mutiny.
Bob Worcester of pollsters MORI said Blair remains favourite to win a third term at next year's election but with a majority of 60-80 in the 659-seat parliament, down from 161 now.
That could leave him at the mercy of a hardcore of Labour MPs so opposed to the war they will now fight him on any front.
"Until the boil of the truth about Iraq is lanced, the prime minister can never put this behind him," Labour MP Jeremy Corbyn told Reuters.
On Wednesday, state prosecutors said they had insufficient evidence to prove 29-year-old translator Katharine Gun broke the Official Secrets Act although she freely admitted leaking a memo which she said revealed a US plot to spy on UN missions.
UN officials reacted angrily to the
claim that Annan was spied on
At the time of the memo, Britain and the US were desperately trying to persuade wavering members of the Security Council to back war.
Top lawyer kills case
Blair's opponents believe the government's top lawyer, Attorney-General Lord Goldsmith, killed the case for fear that questions about the war's legality would be raised.
Goldsmith denied the case was dropped for political reasons, or that he took the decision.
"At the time we started military action it was my own considered...view that military action was lawful," he told parliament. "I believe today it was the correct legal position."
Blair's spokesman said the government would review the Official Secrets Act to see if it needed amending.
Gun, who worked at surveillance centre GCHQ, said she had exposed "illegality and wrongdoing on the part of the US government, who attempted to subvert our own security services".
The Observer newspaper said the leaked memo showed America asked for Britain's help to bug the offices of Security Council members - Chile, Mexico, Cameroon, Angola, Guinea and Pakistan.
But some diplomats were philosophical about the affair. Spain's UN ambassador Innocencio Arias said: "Everybody spies on everybody."