Alvaro Uribe, the president, said he ordered the arrests of the leadership of the United Self-Defence Forces of Colombia, or AUC, "so the peace process can go forward and gain credibility".
On Monday, Uribe ordered AUC commanders to report immediately to detention centres to be questioned by investigators for their role in some of the worst atrocities in the South American nation's 40-year-old civil war.
The paramilitary leaders had agreed to be questioned on the rights abuses as part of the peace accord.
He said that if they did not comply, they faced losing benefits, such as the suspension of extradition orders.
Three militia leaders were arrested early on Wednesday at the group's headquarters in the western city of Medellin, Carlos Restrepo, an AUC spokesman, said.
Other AUC leaders were arrested throughout the country.
Salvatore Mancuso, a senior AUC commander, escorted by his bodyguards, turned himself over to the authorities at a police station in Monteria, telling journalists that the paramilitaries "are respectful of the president and the promises we've made to Colombia".
Salvatore Mancusso (L), an AUC
leader, turned himself in
Since 2003, when they began disarming, the paramilitary leaders have been allowed to remain out of jail, free to spend their sizable fortunes and give interviews to journalists in violation of their promise to not meddle in politics.
Appeasing the US
Mauricio Silva, a political analyst, said: "This looks like a slap with a ruler to appease the United States. There's no real domestic pressure on Uribe to explain these detentions."
The United States has requested the extradition of several AUC leaders on charges of being among the country's biggest drug traffickers. Karen Tandy, head of the US Drug Enforcement Administration, met Uribe in Bogota on Monday.
Sabas Pretelt, the Colombian interior minister, said militia commanders rounded up would be taken to an austere and safe detention centre.
"They're not going to be extradited, but they have to start complying with soberness what they've agreed to," he said.
AUC leaders threatened to walk out of the peace accord after the constitutional court in May overturned key components of the law governing their demobilisation, requiring them instead to confess their crimes fully and forfeit assets - acquired legally or illegally - to compensate victims' families.
The court also ruled that failure to comply could result in a life behind bars, even though the government promised maximum sentences of eight years.
The AUC insists that it has kept its word - despite evidence collected by the United Nations and human rights groups that some demobilised leaders are rearming and others are running criminal organisations.
The peace agreement has led to the demobilisation of more than 30,000 paramilitary fighters in the past two years and has contributed to a recent sharp drop in violence.
Colombia's paramilitary groups were created in the 1980s in response to marauding leftist rebels in the countryside.