Opponents have also criticised Abdullah Gul, the Turkish president, and former member of AKP, for approving the law in just a few days.
Gul's office said a review of the law had started before the draft went to parliament.
It was adopted by parliament on December 1 and put into law three days later.
"They do not know the importance of judicial independence ... but we will explain it to them, we will teach them," Ozdemir Ozok, a Turkish Bar Association Chairman shouted to the crowd in Ankara.
The judiciary is traditionally a bastion of secularism and the march turned into a pro-secularist rally.
"Turkey is secular and will remain secular," protesters shouted, waving Turkish flags and pictures of Mustafa Kemal Ataturk, founder of the modern secular republic.
No official figures were available, but television footage showed between 5,000 and 10,000 people at the march which was held amid tight security.
The AK party is preparing a constitution which will replace the text drafted after a 1980 military coup, but secularists fear the new text will blur the separation of religion and state.
The demonstrations come after the AK party signalled last week that the new charter would ease a ban on the use of Muslim headscarves at universities, a move opposed by the secularist establishment.
The march follows earlier demonstrations this year where protesters accused the AK party, whose roots are in political Islam, of undermining Turkey's official secular order.
However, the AK says that it has drawn a line on its Islamist past and is moving towards a conservative democratic stance.