Al Jazeera's Mike Hanna, reporting from Johannesburg, said the group was now trying to continue its efforts from neighbouring South Africa with a series of meetings with various parties, including the president of Botswana and Zimbabwe's opposition leader, Morgan Tsvangirai.
|Annan said Zimbabwe made it "very clear" that his delegation would not get visas
On Sunday, the three Elders talked to men, women and children who have fled Zimbabwe and sought refugee at the Central Methodist church in downtown Johannesburg.
Over 1,600 people are squeezed into the church with many more sleeping on the pavement outside. Over recent months there has been an increasing number of children making their way to the church.
The Zimbabwean government denied barring Annan from entering the country, saying the visit was simply postponed.
Simbarashe Mumbengegwi, Zimbabwe's foreign ministry spokesman, said "the postponement was necessary because Mr Annan had made no prior consultations with the government of Zimbabwe regarding both the timing and programme of his proposed visit, as is the normal practice".
"It is most unfortunate that the former secretary-general has, for reasons best known to himself, misrepresented the position of the government of Zimbabwe," he added.
Christopher Mutsvanga, a spokesperson for Zimbabwe's governing Zanu-PF, told Al Jazeera that the Elders made no effort to speak to the Zimbabwean government in time to "make an arrangement" for the proposed visit.
"I don't know under what international convention they govern themselves. One needs to understand who they are and what they stand for and what they are up to," he said.
"Zimbabwe's political problems are now being dealt with under SADC [Southern African Development Community] with President [Thabo] Mbeki as the mediator. And he has ample authority to deal with them.
"We do not know under whose auspices these guys are trying to come to Zimbabwe. We need to understand that."
Mutsvangwa said that while Carter was welcome as he has "an impeccable standing in Zimbabwe", Annan did nothing to help Zimbabwe's cause.
"We do have some problems with Kofi Annan. He had ample authority to deal with the Zimbabwe situation, and was invited to come to Zimbabwe but he never did for his own reasons," he said.
"He never condemned the illegal sanctions on Zimbabwe. And he left the UN without ever having been on record for criticising the sanctions. He has been completely silent on the plight that Zimbabwe finds itself in today. He is shedding crocodile tears."
'Urgent response needed'
On the Elders' website, Annan said "food shortages, a lack of seed and fertiliser for planting and the breakdown in health services are all having a serious effect on the people".
"We understand that the situation requires an urgent response and that delays will only prolong the people's suffering," he added.
On Friday, the World Health Organisation (WHO) reported that nearly 300 people had recently died from a cholera outbreak.
Fadela Chaib, a WHO spokesperson, said 6,072 cases had been reported between the start of August and November 18, with a rise in the number of reported cases in the past two weeks.
Doctors and medical staff had protested in Harare, the capital, earlier in the week, saying the healthcare system is suffering from severe shortages.