The text urges UN member states to impose sanctions on North Korea, including tougher inspections of cargo, a tighter arms embargo and new financial restrictions.
Susan Rice, the US ambassador to the UN, praised the council's move as "unprecedented", but warned that North Korea could react to the new sanctions with "further provocation".
"Based on past experience and a pattern that North Korea has of reckless and dangerous actions, it would not be a surprise if North Korea reacted to this very tough sanctions regime in a fashion that would be further provocation," she said.
Zhang Yesui, China's UN ambassador, said the resolution showed the "firm opposition" of the international community to North Korea's nuclear ambitions.
North Korea's nuclear test last month defied a previous Security Council resolution adopted after the North's first underground nuclear test in October 2006.
Japan is expected to impose its own sanctions on North Korea, including suspending all trade, in a largely symbolic demonstration of its opposition to the test, the Kyodo news agency reported.
Al Jazeera's Tony Cheng, reporting from North Korea's neighbour, China, said that it was difficult to judge what effect the new sanctions would have on the already impoverished state.
"This is really just tightening sanctions that already exist on North Korea, but they do target it in specific areas," he said.
"I think that one area that will hurt quite a lot will be the ban on conventional weapons arms sales and the possibility of stopping ships going to and from North Korea ... that is a business that could earn Pyongyang as much as $100m."
Jamie Metzl, the executive vice-president of the Asia Society, said North Korea had exported arms to about 20 countries in the past, including Iran, Egypt, Pakistan, Myanmar, Zimbabwe and Sudan.
"Their finances are in big trouble. They have almost nothing that anybody else wants to buy but these arms," Metzl said.
The backing of China, one of North Korea's key trading partners and regional allies, and Russia for the resolution gave greater weight to the new sanctions as they have been reluctant to act in the past.
"To a certain extent, China has been happy to leave North Korea to its own devices," Cheng said.
"Now China is profoundly concerned about the regime in Pyongyang, which seems increasingly unstable and seems increasingly not to follow Beijing's lead."
The UN vote comes amid continuing tensions on the Korean peninsula after North Korea on Thursday demanded a 3,000 per cent increase in rent and a 400 per cent increase in wages for 40,000 workers employed by South Korean companies at an industrial park in the North Korean border town of Kaesong.
North Korean state media issued a statement on Thursday saying that relations between the two countries had reached the "phase of catastrophe" and that the Kaesong complex had been "thrown into a serious crisis".