Rights activists, being detained in a building where the meeting was being held, said Sunday's crackdown called into question Sudan's right to host the AU summit.
Usman Hummaida, from the Sudanese Organisation Against Torture (SOAT), said: "Towards the end of the meeting a group of security men came and demanded to see the agenda, the list of participants and our recommendations.
"Everyone is being detained and we have been asked not to talk on the phone. We have not been told why we are being held," he said. The meeting was to discuss closer co-operation with the AU on human rights issues.
Representatives of Amnesty International, Anti-Slavery International and the International Bar Association were among those being held, Hummaida said.
Mudawi Ibrahim Adam, a Sudanese human rights activist who has previously been arrested by the government, said: "They are harassing people and trying to get all the laptops from them.
"They cannot be hosting a summit while they have this kind of conflict and they cannot be the chairperson of the African Union."
the European Union ambassador in Khartoum, said the police action was a step backwards for Sudan, where freedom of movement and press had improved recently.
"Everyone is being detained and we have been asked not to talk on the phone. We have not been told why we are being held"
Sudanese Organisation Against Torture
He said he was worried for his staff who were inside the meeting. Some of the foreign delegates were allowed to leave but many stayed, saying they would not leave until the Sudanese were also released.
Journalists were told by police and state security officers to leave the building, next door to the Ministry for Humanitarian Affairs, and one had his recording equipment seized, a witness said.
Sudan, facing accusations of human rights abuses, is hosting the AU summit for the first time and is trying to persuade African leaders to back its attempt to be president of the 53-member body.
The action by the security forces took place hours after Sudan signed up to an AU peer review mechanism, designed to reinforce good governance in Africa.
Khartoum's attempt to chair the AU has provoked criticism from rights groups, which say it would make a mockery of AU efforts to bring peace to Sudan's Darfur region.