The men were on Tuesday declared persona non grata amid claims they had made contact with Taliban fighters in Helmand province.
Several Afghan colleagues of the two diplomats were also arrested.
The UN insists the affair is a result of a misunderstanding arising from a visit the men made to the southern town of Musa Qala, recently recaptured by US-led forces after 10 months under Taliban control.
But an Afghan official said on the condition of anonymity: "It is the government's last decision."
The Afghan official said: "Not only did they hold talks with the Taliban, but also had given them money.
"It is not clear whether they were supporting the insurgency or not."
Siddique, the UN spokesman, acknowledged that the men visited Musa Qala but said: "We do not talk to the Taliban - full stop. That is not what we were in Helmand province to do."
He said the men assessed the "stabilisation" efforts after the military offensive, and spoke to a number of locals including "people who are perhaps undecided whether they are supportive of the government of Afghanistan".
"We have subsequently been informed that our presence in Helmand was detrimental to national security interests. We disagree with this assessment."