As delegates gathered for the inaugural meeting on Monday, an AU official said five heads of state had met Sudanese President Omar Hassan al-Bashir and told him "there was a consensus that he should withdraw" Sudan's bid.
The official did not name the countries but confirmed that one of them was Nigeria.
Sudan's bid to take over the AU chairmanship is the only one put forward so far but has African nations deeply divided.
Human rights groups say awarding leadership of the pan-African body to Sudan would be akin to rewarding a regime that has been accused of practising genocide in Darfur.
Darfur rebels taking part in AU-sponsored talks in Abuja, Nigeria, have said they will pull out of negotiations if Sudan wins the presidency. They say it will undermine the AU's ability to act as an impartial arbitrator in the dispute.
On the eve of the conference al-Bashir said: "The essential issue is not the chairmanship but rather the success of the summit. This is our priority."
Traditionally the host of the AU summit has taken over as chair, but the process is not formalised.
Many Africans nations are wary
of Sudan's bid for AU leadership
On Sunday Sudanese authorities detained 50 local and international human rights activists meeting on the sidelines of the AU summit.
Campaigners said the arrests further highlighted concerns over Sudan's possible presidency of the group.
About 300,000 people have been killed in the western Sudanese region of Darfur since 2003.
A further two million have been displaced by the fighting between rebels and militia-backed government forces.
On Sunday the head of the UN mission in Sudan, Jan Pronk said: "As far as the security on the ground is concerned, there is chaos, in particular in west Darfur where there are many parties fighting. There are still attacks by militias on civilians."
Many African countries are reluctant to back Sudan's bid for the AU chair, with Congo emerging as an alternative candidate.
Speaking at a news conference in Brazzaville, the Congolese prime minister Isidore Mvouba said it was "not impossible" that President Sassou Nguesso takes over the chair of the AU, currently held by Nigeria.
Mvouba said that the Congolese president was "being courted by several of his counterparts asking him to take over the chairmanship."
Aside from the AU chairmanship, other issues on the summit agenda in Khartoum include the current situation in Ivory Coast where four days of anti-UN protests have provoked the Security Council to consider sanctions.
The summit will discuss the
current situation in Ivory Coast
Ivorian president Laurent Gbagbo is not attending the summit.
African leaders will also be asked by Senegal to extradite Chad's former ruler Hissene Habre to Beligum to face trial for crimes against humanity.
The AU was founded in 2002, succeeding the Organisation of African Unity, and is modelled loosely on the European Union.
It aims to help to promote democracy and human rights as well as combatting Africa's HIV epidemic, although it is often embroiled in Africa's many armed conflicts and has struggled to impose itself as a viable pan-African alternative to the UN.
Sudan's campaign to win the AU chair came as the body was seeking to reinforce its troubled 7000-strong peacekeeping force in Darfur with help from the United Nations.