Tens of thousands of people have fled Myanmar's Rakhine state since the military began a security operation last October in response to what it says was an attack by Rohingya armed men on border posts, in which nine police officers were killed.
A February UN report said the operation targeting the Rohingya involved mass rapes and killings, possibly amounting to crimes against humanity.
The independent, international mission should be dispatched "urgently" to establish the facts of the alleged atrocities, the UN Human Rights Council decided in Geneva on Friday, in a resolution adopted by consensus.
The investigation would be launched "with a view to ensuring full accountability for perpetrators and justice for victims," the resolution said.
The investigators must provide an oral update in September and a full report by this time next year, said the resolution backed by the European Union. Some countries, including China, India and Cuba, dissociated themselves from the resolution.
But the council stopped short of calling for a Commission of Inquiry - the world body's highest level investigation - into the violence, despite a call by Yanghee Lee, the UN's special rapporteur on rights in Myanmar.
Earlier this month, Lee told reporters that European leaders wanted to give Myanmar's new civilian government led by Nobel Peace laureate Aung San Suu Kyi more time before launching a top-level inquiry.
Myanmar has been staunchly opposed to a Commission of Inquiry. The country's rights council ambassador, Htin Lynn, also called the move to send a lower level investigation "unacceptable".
Myanmar has launched its own domestic inquiry into possible crimes in Rahkine and appointed former UN chief Kofi Annan to head a commission responsible for healing long-simmering divisions between Buddhists and Muslims.
"Let the Myanmar people choose the best and the most effective course of action to address the challenges in Myanmar," Lynn said, referring to Friday's resolution.
"We will be doing what needs to be done and we will do it with great prudence and probity," he added.
Activists welcomed what they called a "landmark decision" by the 47-member forum, while regretting that it was not a full international commission of inquiry, and called on the government to cooperate.
"It is unfortunate that the government of Burma/Myanmar has chosen to disassociate itself from this resolution," John Samuel, executive director of FORUM-ASIA, said in a statement.
|Infographic: Where Rohingya are fleeing too? [Al Jazeera]|
Source: Al Jazeera and news agencies