This is according to exit polls.
As Guatemalans awaited official results on Monday, El Periodico daily published an exit poll placing opposition candidates Oscar Berger and Alvaro Colom in first and second place with 42% and 27% of the vote respectively.
Rios Montt came in third with 15% of the vote in the impoverished Latin American country, excluding him from a 28 December runoff race.
"The exit poll surveyed 1273 voters at 68 polling stations, and had a 3.8% majority positive that voter turnout was so high on this election day and all the fears were unfounded," said Tom Koenigs, head of the UN Verification Mission in Guatemala.
"Up to now, in general terms, the day was peaceful... there were problems here and there, but Guatemalans were patient," he said.
He described Sunday's vote as "the most watched elections in the history of the country," with more than 5000 observers on hand led by contingents from the Organization of American States and the European Union.
In the capital, people lined up outside polling stations before dawn, waiting for the doors to open. There was a heavy turnout of the country's five million voters throughout the day, and by the time the polls closed, officials said those still in line would be allowed to vote.
Rural areas also reported a heavy turnout, with a strong contingent of indigenous voters at the ballot box.
In Abou village of Chajul, the crowd waiting to vote turned violent when their polling station was late in opening, rescue workers said.
Another vote-related fatality occurred when an official with a centre-right party was shot in the capital, in what authorities said appeared to be a criminal case.
And ballot boxes were set ablaze at two remote polling stations over a dispute among local candidates, officials said.
Voters were also choosing a vice president and replacing all 158 members of the unicameral Congress, 331 mayors and Guatemala's representatives to the Central American Parliament.
The build-up to the election was plagued by rumours that supporters of Rios Montt would turn violent if he lost the race after they staged violent protests during the campaign that left three people dead.
The election was marked by an
unexpected high voter turnout
The 77-year-old former general and current president of Congress insisted he would respect the outcome of the poll.
"I'm an important figure, I'm president of Congress, and it would be absurd to think otherwise," he said as he cast his ballot on the outskirts of Guatemala City.
Rios Montt ran the country for 18 months from 1982 to 1983, at the height of the 1960-1996 civil war during which about 150,000 people were killed in internal fighting and another 50,000 went missing.
He has been widely condemned for his "scorched earth" policy, which wiped out entire towns suspected of supporting or harbouring leftists. A UN-backed commission has accused him of genocide.
But his presidential attempt was backed by poor voters attracted by his social programmes and tough law-and-order platform.
Rios Montt had tried twice before to run for president, but each time was disqualified by a clause in the 1985 Constitution that bans former coup leaders from the presidency.
In July, the Supreme Court, with members of his ruling Republican Front party on the bench, ruled the ban could not be enforced retroactively.
Guatemala has a population of 11.2 million, 80% of whom live in poverty.