The move means Norway can no longer play a role as one of four mediators, alongside the US, Britain and Japan, in Sri Lanka's peace talks.
"We cannot be facilitators in a peace process which has in effect been suspended since 2006," Jon Hanssen-Bauer, Norway's special envoy to Sri Lanka, said.
Demonstartors demanding Norway help bring an end to the conflict in Sri Lanka, broke into the Sri Lankan embassy in Oslo on Sunday, smashing windows and furniture.
No one was hurt in the protests, police said, and the demonstrators, who had videotaped themselves and posted the attack on the internet, were not arrested.
Sri Lanka's foreign ministry said Norway had shown "sheer neglect" of its obligations by failing to protect the embassy despite repeated requests for better security.
Protests have been on the rise in numerous world cities where there are large Tamil populations.
"In these circumstances, the government of Sri Lanka perceives that it is no longer feasible for Norway to act as facilitator in the peace process," a statement from Sri Lanka's foreign ministry said.
Sri Lanka formally invited the Scandinavian nation to act as a mediator in the conflict in January 2000; Norwegian officials managed to secure a ceasefire which came into force in February 2002.
But the Sri Lankan government pulled out of the truce in January last year, accusing the LTTE of frequent ceasefire violations and of using the break in fighting to rearm.