Major Hans-Jakob Reichen, a UN military spokesman, said on Sunday that 35 rebels were killed in the operation, which was continuing.
An Indian peacekeeper was killed and four others injured, and three Congolese government forces were killed and 16 injured.
Since last week's referendum, which paves the way for elections by the end of June 2006, UN and Congolese forces have tackled militias blamed for undermining the peace process and raping and murdering civilians in eastern areas.
Reichen said the latest fighting began on Saturday about 30km south of Beni in North Kivu district after fighters of the Ugandan Allied Democratic Forces/National Army for the Liberation of Uganda (ADF/NALU) ignored an ultimatum to negotiate voluntary repatriation.
Uganda and neighbouring Rwanda backed rebels in Congo's complex five-year war and nationals from both countries still operate in militia groups across the east of Congo.
On Saturday about 300 Nepalese UN peacekeepers and 1500 government troops backed by helicopter gunships attacked and captured Nioka, 80km northeast of Bunia, the main town in the unruly Ituri district of northeastern Congo.
UN and Congolese forces have been fighting in the area for several days against an ethnic Lendu militia, which has refused to join a UN-backed disarmament process.
Dozens of militia fighters and a number of Congolese troops have been killed since last Sunday's vote, the first free national poll for 40 years in Africa's third-biggest state.
Nearly four million people have died since war broke out in 1998, mostly from war-related hunger and disease.
Latest results announced on Saturday showed that, with votes from nearly three-quarters of polling stations counted, the "Yes" vote had 83%, with just under 17% for "No".
"No" campaigners say they will contest the result in the Supreme Court on the grounds that the government's "Yes" campaign was unfairly supported by the international community, which has a role as sponsor and guarantor of a peace deal agreed in 2003.
The referendum won a verdict of "free and fair" from European Union observers despite some violence and intimidation at some polling stations. But it is still unclear how the vote went in some areas where militia groups operate.
UN troops and a new Congolese army drawn from the ranks of government and rebel forces aim to establish central government control in areas dominated for years by foreign-backed rebel factions or local militias.
Last month a delegation of UN Security Council ambassadors visited Congo and its neighbours and backed Ugandan calls for more robust action to dislodge the ADF/NALU and a number of other Ugandan rebel groups from bases in northeast Congo.
The United Nations estimates that about 2000 armed Ugandans operate in Congo, for a confusing array of foreign armed militias left after years of fighting driven in part by competition to control Congo's rich mineral resources.