Dev Gurung, a Maoist cabinet minister, said: "We have submitted our resignations to the prime minister... because the talks were not successful."
Political parties had attempted to convince the Maoists on Monday to remain in the administration and to call off the planned demonstrations.
The Maoists have been insisting the nation must be declared a republic ahead of the November 22 vote, insisting that King Gyanendra and his supporters were trying to sabotage the election.
Now they say they will launch street protests to ensure the election is held on time but will keep to the ceasefire.
The Maoists have called a rally in the capital Kathmandu later on Tuesday when they are expected to announce their campaign.
Some analysts said the Maoists, who entered mainstream politics only recently, could be nervous about the elections and are trying to delay the vote.
Lok Raj Baral, head of the independent think-tank Nepal Centre for Strategic Studies, said there was "a widespread feeling that the Maoists have less chance of winning as many seats as will be wrested by other main parties".
He said the Maoists want to delay the polls so that they are better organised.
The new government has already stripped the monarch of almost all his powers, including his control over the army.
The conflict between the government and Maoists that began in 1996 killed around 13,000 people and hit the aid- and tourism-dependent economy of Nepal, one of the world's poorest nations.