The Dutch government said it was "extremely disappointed" at the decision and called for an explanation.
"The Netherlands maintains that Iceland is compelled to pay back the money," Ruud Slotboom, a Dutch finance ministry spokesman, told the AFP news agency.
"The Netherlands maintains that Iceland is compelled to pay back the money"
Dutch finance ministry spokesman
"We expect of the government of Iceland to give us an explanation in the short term of the situation now created and the steps to be taken."
The compensation bill has sparked anger in Iceland, which was hit by a financial meltdown in October 2008.
About 60,000 people - about one-quarter of the country's electorate - have signed a petition protesting against the bill and calling for the issue to be put to a referendum.
Al Jazeera's Jonah Hull, reporting from London, said "the domestic distate for this bill stems from the feeling among many Icelanders that they don't want as taxpayers to be paying for the sins of their banks.
"They feel that their country is in no position .. to pay for the £5bn to Britain and the Netherlands."
Analysts said that Grimsson's rejection of the unpopular bill put aid from international lenders at risk.
"You can't hold a referendum in three days and this issue is pressing. The IMF will have to put on hold payment of any future tranches of aid until we have a 'yes' vote," Lars Christensen, and analyst at Danske Bank, said.
A Finnish official said the decision was likely to delay a loan of $2.6bn from Nordic countries.
It could also put Iceland's aspirations to join the European Union in serious jeopardy, with Britain and the Netherlands having veto power over the membership bid.