The demonstration comes after Ahmet Necdet Sezer, Turkey's pro-secular president, told military officers in an address on Friday that "for the first time, the pillars of the secular republic are being openly questioned".
Sezer steps down on May 16 and parliament will elect the new president early next month.
Erdogan's Justice and Development party, which dominates parliament, is expected to announce its candidate this month to replace Sezer in the appointed presidency.
He is due to declare whether he will run after a meeting with his party on Wednesday. If he runs, the party is expected to select him as president.
Another pro-Islamist official could then be selected for the premiership, placing the executive branch entirely under the control of the Islamic-leaning ruling party.
Secularists in the 99 per cent Muslim country fear that if Erdogan, or someone close to him, wins the presidency next month, the government will be able to implement an Islamic agenda without opposition.
|Protesters gathered in Ankara|
on Saturday [EPA]
Erdogan's government has denied it has an Islamic agenda.
However, his party was elected to an overwhelming majority in parliament and can appoint whoever it wants to the presidency.
Erdogan has pushed Turkey closer to the European Union, but he is also known for his religious principles.
He has stoked secularist concerns in Turkey by speaking out against restrictions on wearing Islamic-style head scarves in government offices and schools and taking steps to bolster religious institutions.
He also tried to criminalise adultery, but came under criticism for the move and backed down under pressure from the EU.
But the country's pro-secular military maintains a strong influence over Turkish politics and in 1997 generals pressured Erdogan's mentor out of the prime minister's office because he was viewed as excessively religious.
Analysts warn that any serious tensions between the government and the military could have a serious effect on the country's economy.
General Yasar Buyukanit, the chief of the military, said on Thursday, in a statement widely interpreted as a warning to Erdogan not to run: "We hope that someone who is loyal to the principles of the republic, not just in words but in essence, is elected president."
Saturday's rally was organised by Sener Eruygur, president of the Ataturk Thought Association and a former commander of Turkey's paramilitary forces.