Hun Sen confirmed on Monday that the cases will not be dismissed, despite growing international pressure over a clampdown on politicians and media perceived as critical of the government.
He said the courts had already begun investigating the cases and it would not be possible to halt the legal process.
Defamation charges were ordered to be dropped last week against Kem Sokha and Pa Nguon Teang of the US-backed Cambodian Centre for Human Rights, as well as against a journalist, Mam Sonando, and a union leader, Rong Chhun.
All had been arrested for their opposition to a border agreement with Vietnam, which critics say cedes too much territory to Cambodia's eastern neighbour.
The unexpected move to release the four initially drew praise from the US and United Nations, but a Cambodian judge and the government's lawyers said last week the cases would go forward.
Hun Sen suggested that his critics, who were released on bail earlier in January, could avoid court as long as they did not further inflame political tensions.
"My suggestion is to ... just postpone [court proceedings] to keep the situation quiet," he said. However, he also said: "If you are rude, the court will summon you, so there will be another problem."
Hun Sen was speaking at a university graduation ceremony where he also replied to foreign criticism of the arrests, saying he had the right to protect himself from his critics.
"Every person has the right to ask the law to protect him when he is defamed," Hun Sen said. "Please, outsider, before you talk, please see the law clearly."
His comments came days after the US Senate passed a resolution demanding that his government "immediately cease and desist from its systematic campaign to undermine democracy, the rule of law, and human rights".
Defamation is a criminal offense in Cambodia and has been used to arrest nearly a dozen activists and political rivals during the last year.
They include Sam Rainsy, the opposition leader who has fled the country and was sentenced in absentia to 18 months in prison.
"Every person has the right to ask the law to protect him when he is defamed"
Hun Sen, prime minister of Cambodia
The United States has been the most vocal foreign critic of recent arrests, which Washington says are part of an attempt by the government to use the courts to crush dissent.
The Senate resolution includes a long list of incidents which allegedly show that Hun Sen "blatantly violated basic democratic principles" - including the jailing of Cheam Channy, an opposition MP, for seven years and the assassination of Chea Vichea, a union leader.