"It was serious fighting and 38 militiamen were killed," a UN source said.
The clash came a day after a UN deadline expired for voluntary disarmament by militias, who have killed hundreds of civilians in the mineral-rich region.
Earlier, the UN mission said South African, Bangladeshi and Pakistani peacekeepers exchanged fire with armed men as they searched two militia camps 40km southwest of Bunia, capital of Ituri.
The UN now has 16,700 soldiers in Congo, making it the largest peacekeeping mission in the world.
It has been criticised in the past for failing to rein in the brutal militias, who have carried out attacks on civilians, kidnapping, torturing and decapitating their victims.
But after nine Bangladeshi peacekeepers were killed in February - the worst single loss suffered by the peacekeeping mission since it began in 1999 - the UN force stepped up military operations against armed gangs.
The military source said many weapons and lots of ammunition were seized in Saturday's raid.
The Congo mission is the largest
peacekeeping force in the world
"The provisional information we have is that 18 militiamen were killed in today's operation," Kemal Saiki, a spokesman for the UN mission, said. He said he was checking for further details of the operation.
"Our troops do not fire unless fired upon, so the militia obviously shot first," he added.
Ethnic warfare has killed more than 60,000 people in Ituri since 1999. Fighting between Hema and Lendu militias has intensified since December, hundreds have been killed and about 100,000 have fled their homes.
The UN mission had given militias in Ituri until 1 April to hand over weapons, and promised to strike at those who failed to comply in a lawless frontierland where hundreds of civilians have been killed this year.
More than 8000 of the estimated 15,000 fighters in the region have surrendered their weapons.
"This is part of the response since the deadline ran out," a UN military source said. "Operations will continue for a while to disarm those who refuse to join the process."
The UN says Congo is the world's
worst humanitarian disaster
"The voluntary disarmament process is over," UN spokeswoman Rachel Eklou said. She said 8200 militia members had officially disarmed, while a few hundred more had also handed in their guns, but had not yet been registered.
Some fighters have said they will not join the disarmament process and Eklou said they would have to face the consequences.
"We will make their lives more difficult - they will be prosecuted or face more military operations," she said.
The UN says eastern Congo is the world's worst humanitarian disaster. This week, aid workers reported a cholera outbreak in three camps set up for fleeing civilians.
Three people have died from the water-borne disease in the past week and there are at least 110 other cases in the camps, which house more than 50,000 people, Unicef said on Friday.
The fighting in the east has cast a shadow over elections due to be held in the former Zaire in June and meant to mark an end to a five-year war that drew in six neighbouring countries before it was declared over in 2003.