The Bundestag lower house defeated the measure on Friday with the help of deputies from Schroeder's coalition of Social Democrats and Greens who followed the chancellor's call to abstain in his effort to bring the general election forward by 12 months.
In a speech before the confidence vote, the embattled leader said his painful economic reform drive had cost him crucial support that he now hoped to regain.
"Without a new mandate, it is impossible to continue with my policies," Schroeder told the chamber.
"If we want to continue with the reforms, and we must, we need to seek a new mandate through elections."
The 61-year-old Schroeder, who wore a dark suit and struck a somber tone, said the "painful" loss of a May election in Germany's most populous state had made it clear that his governing centre-left coalition was at risk.
President Horst Koehler now has 21 days to decide whether to dissolve parliament and order new elections, which will probably take place on 18 September.
President Koehler (L) will decide
whether to dissolve parliament
Schroeder's Social Democratic Party (SPD), beleaguered by the anaemic economy and around 12% unemployment, is trailing the conservative opposition Christian Union bloc (CDU/CSU) by about 20 percentage points in most polls.
A conservative victory would see opposition leader Angela Merkel become Germany's first woman chancellor.
In a speech that often sounded like a farewell, Schroeder said that the unpopular economic and labour market reforms his government introduced since it took power in 1998 had been bold and essential steps that his predecessor Helmut Kohl had failed to undertake.
"The reform process is unique in the history of the federal republic in its scope and consequences," Schroeder said.
"We tackled what our predecessors neglected. We started what the CDU and the FDP never had the courage to do in 16 years in government," he said, referring to the conservatives' liberal coalition partners, the Free Democrats.
Schroeder said that his SPD had paid the political price for those reforms, known as Agenda 2010, but was ready to battle to win back Germany's voters.
"Agenda 2010 led to fights between the parties and within the parties and - I do not want to hide it - my party suffered in particular," he said.
Germans are angry that Schroeder has failed to keep his seven-year-old promise to drive down unemployment and that they now in fact have more reason to fear for their jobs than at any other time since World War II.
Angela Merkel: The opposition
parties are ready for challenge
But many left-wing SPD deputies have publicly questioned Schroeder's course, saying that measures such as cuts to unemployment benefits put an unfair burden on the poor.
Merkel, who delivered a rebuttal after Schroeder's speech, said that the government had presented an incoherent and chaotic response to the major problems facing the country.
She said the Christian Union parties were ready to confront those challenges.
"The next election will be an election about the direction the country is taking. It is not enough to say we'll carry on as before," Merkel said to applause.
Although the new elections are likely to take place in September, 12 months ahead of schedule, the decision is far from certain.
A poll by independent opinion research institute Forsa this week showed Merkel's Christian Union alliance with 47%support, versus 26% for Schroeder's Social Democrats (SPD).
Their junior coalition partner, the Greens, tallied 7%.
Friday's vote was the second that
Schroeder has faced
Meanwhile a new leftist alliance grouping former East German communists and SPD dissidents reached 11% support, creating a fresh complication for Schroeder.
The Christian Union's preferred coalition partner, the Free Democrats, scored 6%.
Although a vast majority of Germans hope the poll will be held this year - one year before schedule - the move could still be thwarted by a court challenge.
Some coalition lawmakers, fearful of losing the election and bewildered by the request to express no-confidence in a government they still support, have said they are weighing an appeal to the federal constitutional court, Germany's highest tribunal.