The government has banned protests in Punjab, the country's most important political constituency, and Sindh, where Karachi is located.
One lawyer, Rashid Razvi, said: "Our struggle will continue. If we are stopped from joining the long march, then other people will reach Islamabad secretly." Police then pushed him into a prison van.
Some Karachi protesters set light to a minibus, firing off volleys of gunfire and throwing stones, police said.
In Lahore, the capital of Punjab, about 2,000 lawyers, political workers and civil activists also rallied.
The Lahore and Karachi demonstrators are due to arrive in Islamabad, the federal capital, on Monday, where they hope to be joined by thousands of other protesters outside parliament.
Kamal Hyder, Al Jazeera's correspondent in Islamabad, said: "It is a test for the new government, as to whether it will be in a position to give people their democratic rights.
"Across the country there has been a heavy clampdown by the security agencies in spite of the fact that the prime minister said that there would be no problem with the march as long as it is peaceful."
Ghafoor Ahmed, vice-president of Pakistan's biggest religious party, Jamaat-e-Islami, was among those detained in Karachi on Thursday.
|Sharif, left, has criticised Zardari for not reinstating the sacked judges [AFP]
A day earlier, police had rounded up about 300 people, including members of the Pakistan Muslim League-Nawaz (PML-N), Pakistan's main opposition party, in an attempt to stall the rally.
Nawaz Sharif, the leader of the PML-N and a former prime minister, had called for people to "change the destiny of Pakistan" by attending.
The PML-N quit the cabinet last year to protest over the new civilian government's failure to honour a deadline to reinstate Iftikhar Chaudhry, the former supreme court justice, and other judges sacked by Musharraf.
In February, Pakistan's supreme court disqualified Sharif from contesting elections, triggering a power struggle between him and Zardari, two politicians who had briefly come together in the campaign to force Musharraf out.
The ruling forced Sharif's party out of power in Punjab, placing the province under central government control.
But in an apparent concession to Sharif, Yousaf Raza Gilani, Pakistan's prime minister, said on Wednesday that the government wanted federal rule over the province to end.
Whichever party has the sufficient mandate to form the provincial government should take over, he said.
The PML-N has the most support in Punjab, although it does not have a clear majority to run the provincial government alone.
Raja Assad Hameed, of the Nation newspaper, said that many of the protesters are looking for the central government to relinquish its control over the province.
"They are coming to Islamabad to tell Zardari that the mandate in Punjab, the powerhouse of Pakistani politics, should be given back to the legitimate representatives of the people and that the governor's rule should be lifted from Punjab," he said.
"The situation could go anywhere from here; the government has lost its credibility and popularity very prematurely."