Seventeen constituencies had been won by the Maoists out of 29 seats for which results were declared and the party was leading in another 65 constituencies, officials said.
The Nepali Congress had won or was leading in 28 constituencies while the Communist Party of Nepal (Unified Marxist-Leninist) had won or was leading in 22, the officials said.
Smaller parties were winning or leading in the remaining 19 constituencies being counted.
The polls are the climax of a peace process that ended a decade-long insurgency by the Maoists to overthrow the monarchy, which claimed 13,000 lives.
Full results of the impoverished Himalayan nation's polls, a dual first-past-the-post and proportional representation system, will not be known for at least a week.
The Maoists say they are committed to democracy but are still classified as a terrorist organisation by the United States.
"This is the people's victory," hundreds chanted outside a counting centre in Kathmandu, many with the hammer and sickle painted on their faces or with Maoist flags wrapped around their heads.
An elected 601-member special assembly is supposed to write a new constitution and usher in a republic in the impoverished Himalayan state, ending a 240-year-old Hindu monarchy.
'Number one party'
Maoists' leader Prachanda, whose nom de guerre means fierce in Nepali, won a landslide victory in his constituency.
He emerged from a counting centre with his forehead smeared with vermilion and his head swathed in layers of marigold garlands.
"We will create a stronger basis for sustainable peace ... economic development and a federal republic by including all sections and communities of people.
"We will establish greater national unity with all political parties after the election," he said, in a statement signalling his commitment to a coalition government, analysts said.
Kunda Dixit, editor of the weekly Nepali Times, said: "At the rate it is going, the Maoists will be the number one party."
Trends so far suggest that there was an overwhelming desire on the part of mostly young voters for change, peace and development."
There are 240 seats decided on a first-past-the-post basis, with another 335 seats decided by proportional representation and 26 to be nominated by the cabinet.
A Maoist victory would be hard to stomach for Nepal's conservative army, traditionally loyal to the king, as well as for the governments of the United States and India. New Delhi worries it could encourage India's own Maoist rebels.
Prachanda, who wants to be Nepal's first president, insists he wants friendly ties with Washington, New Delhi and Beijing.
"We will have good relations with all countries, including India and China," he said.