Gaddafi was handed the chairman's gavel by Jakaya Kikwete, the Tanzanian president and outgoing AU leader, to applause from other leaders on Monday.
Some members of the AU were, however, said to be uneasy about his nomination.
Gaddafi has long promoted stronger union within the organisation and previously outlined his vision for a continent-wide government.
He has also previously said he wants a single African military force, a single currency and a single passport for Africans to move within the continent.
Speaking to Al Jazeera, Richard Dowden, director of the Royal African Society, said: "This says a lot about what African leaders think of the African Union. It was hoped that it would give great new leadership to Africa, create a sense of pan-Africanism even if they were not going to unite politically.
"Now it has got all these aspirations to be a club of democrats - and this is a man who has been a dictator for 40 years.
"I think it says something about what African leaders think about their own aspirations to create a continent of democracy and transparency and accountability, the sort of things that they would like to aspire to.
Work, not words
In his inaugural speech, Gaddafi said: "I hope my term will be a time of serious work and not just words.
"I shall continue to insist that our sovereign countries work to achieve the United States of Africa."
He admitted that African leaders were "not near to a settlement" on the issue, saying, "We are still independent states."
"It is your decision to respond to the call for unity, to push Africa forward towards the United States of Africa," he said.
In a closed-door debate earlier on Monday, Gaddafi failed to receive backing for the idea of a "United States Of Africa".
Leaders decided instead to consider ways of expanding the mandate of the existing AU Commission, which will be renamed the AU Authority.
Dowden of the Royal African Society said Gaddafi "has a terrible record, certainly in Western Africa, for supporting some very nasty movements in Liberia and Sierra Leone, and stirring up rebellions in other countries under this idea of a revolutionary Africa.
"Now he has switched, he is now trying to court all the kings of Africa, all the old kingdoms."
Last August, a meeting of more than 200 African kings and traditional rulers bestowed the title "king of kings" on Gaddafi.