"It is getting heated between Gaddafi and the southern Africans," one delegate told Reuters as he stepped out of the conference hall.
Gaddafi, describing himself as a soldier for Africa, has been impatient with the slow pace of integration.
Mwai Kibaki, Kenya's president, and member of the gradualist camp, expressed strong support for unity in his speech.
"The advantages of Africa's unification are enormous for our people. ... A unified Africa will have stronger bargaining power," he said.
Tajudeen Abdul Raheem, from the UN Millennium Campaign Group, said: "We have tried everything else, why not unity? "The threat of globalisation has made many of our states irrelevant anyway.
"The Organisation of African Unity was formed in 1963 - unity has always been the destination. In 2002, it was restructured to the African Union - unity is still the agenda."
Kibaki said Africa's eight regional economic communities should be the building blocs of a united continent and their integration should be accelerated.
"Gaddafi's idea is good in theory but it won't work for a very long time"
Van Zwam, Dendermonde, Belgium
Send us your views
He called for a pan-African language to unite the continent and advocated the use of Swahili - spoken in much of eastern Africa - saying its use as a common tongue "will assist in nurturing greater unity and co-operation amongst our people".
Many North Africans have called for greater use of Arabic. Summit leaders have come under criticism for largely ignoring pressing issues like Sudan, Somalia and Zimbabwe at this meeting to concentrate on unifying the continent.
Sceptics point to decades of wars, coups and massacres that often sprang from ethnic and religious fault lines on a continent whose countries were artificially carved up by former colonial rulers.
Alpha Oumar Konare, the AU's top diplomat, supported an integrated continent in his summit opening speech on Sunday, but said many challenges must first be overcome, including the future of existing pan-African bodies and regional economic blocs.
Abdul Raheem said: "Nobody, in the debate so far, has declared themselves officially against union.
No one has even declared themselves against the formation of a unity government. "What they are disagreeing about is the timetable for it … What they are discussing here is really the content of what a union government would look like."