Hasim Kilic, deputy head of the court, told a news conference: "A decision was taken to stop the process."
The court ruled that 367 members of parliament had to be present during voting for it to be valid.
"We have seen this theatre many times... [there is] no way other than democracy"
Srimedya, Bursa, Turkey
Send us your views
Only 361 deputies voted in last Friday's presidential ballot, 357 of them for Gul, the sole candidate.
The court ruled that this number was insufficient and that the small turnout rendered the vote invalid. The court's decision is final and cannot be over-ruled.
Recep Tayyip Erdogan, the prime minister, could now propose a different candidate for the job.
The army has publicly threatened to intervene in politics to prevent an Islamist president from the threatening the country's secular constitution and identity.
The military, which sees itself as the final guarantor of the secular state, has ousted four governments in the past 50 years, most recently in 1997 when it acted against a cabinet in which Gul served.
Barnaby Phillips, Al Jazeera's correspondent in Ankara, said that the verdict was a clear defeat for the ruling party and said that within the AK Party there is disappointment with this decision.
"Senior AK Party officials have told us they did not expect this verdict.
"We have heard that they may go to parliament anyway and try and resume the process of getting Abdullah Gul elected as president."
He also said that "they may be hoping to do a deal with some of the smaller parties - particularly with the Motherland Party which only has 20 members. If they do succeed in winning them over, they may succeed in getting the majority that they need to get Abdullah Gul elected as president.
"They need to win over some 15-20 members of parliament. This may hinge on winning over some of the members of some of the smaller parties."
Turkish financial markets reacted badly to the threat of political instability and recorded their biggest falls in a year on Monday and the currency lost more ground on Tuesday. The lira recovered some ground on the news late in trade.
Ali Babacan, the economy minister, said the economy was ready for early elections, a comment seen as an attempt to calm markets.
The secularist establishment fears if the AK Party secures control of the presidency it will chip away at Turkey's secular system. The party denies the charge.
Parliament, in which the AK Party has a big majority, elects the president for a seven-year term in predominantly Muslim Turkey.
Analysts say early national polls are the only way to defuse the standoff.
Damla Aras, a Turkish political analyst in London, told Al Jazeera that the constitutional court's decision had not come as a surprise.
"In the past few weeks, the army has been giving clear signals that it did not want Abdullah Gul or anybody from the AK Party as the president."
She also felt that a fresh round of national polls, if held, would not be before August.