Panel members attacked Colombo in a statement last month for blocking efforts to ensure that minimum standards were maintained in investigations.
Nigel Rodley, a British panel member, told reporters that the government's tone "was becoming increasingly disrespectful [of the experts]".
"They were accusing us of all kinds of nefarious stuff, including making way for an international panel to monitor the rights situation on the ground here. It's not true, the allegations are baseless," Rodley said.
The panel was formed in 2007 to oversee a presidential commission of inquiry into 16 cases of major human rights abuses in Sri Lanka.
Among the cases was the August 2006 massacre of 17 local aid workers attached to a French charity in the island's northeast.
Evidence has pointed to the involvement of security forces and a state cover-up.
P N Bhagwati, head of the panel and a retired chief justice of India, said: "The commission of inquiry has been slow to respond to our recommendations.
"We don't see a point of carrying on our role further."
Lack of will
Bhagwati also accused Colombo of making little effort to enable the commission to maintain its legal and financial independence in terms of funds to carry out field visits and to hire its own legal counsel, without taking on the state attorney general (AG).
"Hiring the state attorney general to provide counsel is a clear conflict of interest. The AG is an organ of the state, so is the army," Bhagwati said.
The panel also recommended that the government set up a witness protection programme to encourage people to testify.
"We conclude that the government as a whole does not have the necessary political will to implement our recommendations.
"We keep repeating the same recommendations over and over again,"
professor Yozo Yokota of Japan said.
The Sri Lankan government has accused the IIGEP of working to an "international agenda" to force a UN human rights monitoring mission on Sri Lanka, a charge denied by the experts.
Sri Lanka is under increasing pressure to improve its record.
Human rights groups say that more than 1,500 people have disappeared in the past two years.
The Sri Lankan government pulled out of a 2002 truce with Tamil separatists in January and since then violence has escalated.
The Liberation Tigers of Tamil Ealam (LTTE) are fighting a decades-long separatist war against Colombo in which tens of thousands have been killed.