The anti-Damascus opposition that launched the campaign against the government hailed the move but vowed it would not stop protests until Syrian troops were out of Lebanon.
Karami announced the resignation of his government during a stormy parliamentary debate over the murder of al-Hariri, which many blame on his government and its political masters in Syria.
"I announce the resignation of the government over which I had the honour of presiding so that it does not pose an obstacle to what is considered by some as the good of the country," he said.
Ghassan bin Jiddu, chief of Aljazeera's Lebanon bureau, said Karami's resignation came as a surprise to everyone.
The prime minister's declaration was greeted with loud applause in the national assembly, where the opposition had been seeking a vote of no confidence in his four-month-old government.
Outside, where people had massed in a sea of red-and-white Lebanese flags, the news was greeted with cheers as demonstrators set off fireworks and honked car horns.
A crowd of about 60,000 rejoiced
at Karami's resignation
"Your turn will come, Lahud, and yours, Bashar," the demonstrators chanted, referring to Lebanese President Emile Lahud and his Syrian counterpart Bashar al-Asad.
The crowd, estimated at 60,000, had defied a government ban on demonstrations and massed in the heart of Beirut as parliament held the debate on al-Hariri's murder in a huge 14 February bomb blast.
Since the bombing, international pressure led by the United States has mounted on Syria to end its dominance of Lebanese political affairs and pull out its 14,000 troops stationed in the country.
Damascus-backed Lahud accepted Karami's resignation but asked him to stay on a caretaker basis.
However, the immediate future of the government remains uncertain and it is not known whether legislative elections due to be held by the end of May will go ahead.
Aljazeera's bin Jiddu said talks were continuing between Lebanese and Syrian authorities over this issue, while the opposition is calling for the formation of temporary government to oversee elections.
Al-Hariri's killing raised fears of a return to the sectarian strife that ripped the country apart during the 1975-1990 civil war, but the murder served to unite religious and political groups in their calls for an end to Syria's domination.
Syria insisted Karami's resignation was an "internal matter" but added that it hoped to see the formation of a new government that would extricate the country from the "current extremely delicate circumstances".
Al-Asad rejects accusations of
Syria's role in al-Hariri's death
Washington said it was watching developments "with great interest".
"The resignation of the Karami government represents an opportunity for the Lebanese people to have a new government which is truly representative of their country's diversity," said White House spokesman Scott McClellan.
The European Commission appealed for calm and for dialogue among political groups.
Al-Asad, in an interview published on Monday, again rejected accusations that Damascus had a hand in the bomb attack, which killed 18 people.
"If we really killed al-Hariri, that would be political suicide for us. Beyond ethical and human principles, the question is who benefits from the crime? Certainly not Syria," he told the Italian daily La Repubblica.
Washington's number two Middle East point man, David Satterfield, called again on Syria to start pulling out its troops before the elections in Lebanon, in line with a UN Security Council resolution adopted in September.
Resolution 1559 calls for the withdrawal of foreign forces from Lebanon and respect for its sovereignty but a specific reference to Syria was deleted from the final text.
Lebanon announced last Thursday an imminent Syrian military pullback to the Bekaa Valley of eastern Lebanon, but there have been no signs of the redeployment.