Hasim Kilic, deputy head of the court, told a news conference: "A decision was taken to stop the process."
The court ruled that 367 members of parliament had to be present during voting for it to be valid. A total of 361 deputies voted in last Friday's ballot, 357 of them for Gul, the sole candidate. AK Party has around 300 members of parliament
The court ruling was binding.
Tayyip Erdogan, the prime minister, could now propose a different candidate for the job.
Turkey has been rocked by the presidential poll standoff, an army threat to intervene and an anti-government rally of up to one million people.
The army, which sees itself as the final guarantor of the secular state, has ousted four governments in the past 50 years, most recently in 1997 when it acted against a cabinet in which Gul served.
Philip Barnaby, Al Jazeera's correspondent in Ankara, said, "Within the AK party there is disappointment with this decision. Senior AK party officals have told us they did not expect this verdict.
"We have heard that they may go to parliament anyway and try and resume the process of getting Abdullah Gul elected as president."
He said that "they may be hoping to do a deal with some of the smaller parties - particularly with the Motherland Party which only has 20 members. If they do succeed in winning them over, they may succeed in getting the majority that they need to get Abdullah Gul elected as president."
"They need to win over some 15-20 members of parliament. This may hinge on winning over some of the members of some of the smaller parties ", Barnaby said.
Turkish financial markets recorded their biggest falls in a year on Monday and the currency lost more ground on Tuesday. The lira recovered some ground on the news late in trade.
Ali Babacan, the economy minister, said the economy was ready for early elections, a comment seen as an attempt to calm markets.
The secularist establishment fears if the AK party secures control of the presidency it will chip away at Turkey's secular system. The party denies the charge.
Parliament, in which the AK party has a big majority, elects the president for a seven-year term in predominantly Muslim Turkey.
Analysts say early national polls are the only way to defuse the standoff.
Damla Raas, a Turkish political analyst in London, told Al Jazeera that the Constitutional Court's decision had not come as a surprise.
"In the past few weeks, the army has been giving clear signals that it did not want Abdullah Gul or anybody from the AK Party as the president."
She also felt that a fresh round of national polls, if held, would not be before August.