Gayoom, who leads the Dhivehi Rahyithunge Party (DRP), has presided over the Maldives since 1978.
He had allowed the presidential election to go ahead after pro-democracy protests and pressure from other nations, marking a sea change in Maldivian politics. At one time it was illegal to criticise Gayoom.
Voice of Maldives radio and the independent Minivannews.com website cited unofficial results which suggest that Gayoom and Nasheed will go into a run-off election.
Cluster of 1,192 islands scattered across 800km of Indian ocean off southern India
Population 300,000, mostly ethnic Dravidians, Indo-Aryans, Sinhalese and Arabs
Official religion: Islam
Official language: Dhivehi, although English and Arabic are widely spoken
Economy is dependent on tourism with growth averaging seven per cent
Timeline: Maldives political milestones
Hassan Saeed, an Islamic scholar and former attorney-general, and Ghaseem Ibrahim, a businessman, are close behind Nasheed in the vote, according to preliminary results.
The final results are expected later on Thursday.
Gayoom could face a strong challenge from Nasheed in a run-off vote if supporters of the other opposition candidates lend him their allegiance, analysts have said.
"If we do force Gayoom into a second round, then he will be in serious trouble," a spokesman for Nasheed’s Maldivian Democratic Party (MDP), said.
Before the election, the MDP had accused Gayoom of trying to rig the poll and secure a first-round win.
The party had concentrated its campaign in the capital, Male, while Gayoom had toured the country's outlying atolls in an attempt to secure support from conservative voters.
Although the nation is the richest per head of population in south Asia, it faces a housing shortage, rising crime and drug abuse.
Gayoom has said that he is the only individual who can implement changes across the country, but opposition parties have accused him of using corruption and strong-arm tactics to maintain his rule.
Highlighting the differences between Gayoom and the opposition, he has taken legal action against two opposition politicians who say that he stole $40m of tsunami aid.