Parliament has been deadlocked since a June election with left-wing and centre-right parties each controlling 100 seats and Topolanek, the prime minister, was expected to lose the vote since the two sides failed to reach a deal on support.
According to official results, 99 deputies voted on Tuesday against the government in the 200-seat lower house, while 96 voted in favour and five were not present.
"The public wants early elections, the public didn't mind this government," Topolanek said after the vote, urging parliament to move quickly to a resolution of the crisis.
"I think that after we submit our resignation, which could be next week at the earliest, we will start negotiations on a bilateral level and then on some kind of group level [to resolve the deadlock]."
Political turmoil has shaken much of central Europe in recent weeks. Financial markets have been unnerved by anti-government protests in Hungary, and by the Polish government cracking apart.
But the Czech deadlock has failed to hit markets, with investors instead focusing on a strong economy that shows few ill effects.
Analysts said, however, the country needs to ensure reforms to sectors such as healthcare, public finances and pensions remain on track for the adoption of the euro.
President Vaclav has the right to
pick a second prime minister
The crown currency slightly weakened after the vote, easing to 28.280 to the euro from 28.265 just ahead of it.
"Without a clear resolution within one or two months or so, the lasting political deadlock may undermine rating agencies' belief that the country is serious about tackling fiscal problems, which may have negative repercussions," said Viktor Kotlan, chief economist at Ceska Sporitelna.
Vaclav Klaus, the president, now has the right to pick a second prime minister, including Topolanek once again.
But his advisers have said that any candidates must be able to show they have a majority in the lower house before Klaus will nominate them.
They added that the right-wing president is likely to push for a government of experts followed by early elections.
Klaus, currently on a tour of Asian nations, was quoted by the CTK news agency as saying that he would not name a new candidate before local and senate elections, which begin on October 20 and run until October 28.
Jiri Paroubek, former prime minister and Social Democrat leader, has been unable to show he could win a confidence vote, but he said he is still against early elections and wants to explore forming his own government.
"It's now very important who Klaus picks to form a government, if he does. I think he pretty well has to choose Paroubek, who will likely get a chance in the third attempt anyway as the right has failed," said Petr Just, a political scientist at Charles University.
A third candidate is chosen by the house speaker. A Social Democrat currently holds the speaker's post, but was sworn in under an agreement among parties that he would step down before it was his turn to nominate a prime minister.