The study will take 12 to 18 months, covering technical, operational, environmental, social and economic analyses of the proposed smelter.
The smelter, to be called the Sarawak Aluminium Company, would be situated in Similajau and is expected to begin operations by the end of 2010.
It will have an initial production capacity of 550,000 tonnes per year, rising eventually to 1.5 million tonnes.
Oscar Groeneveld, Rio Tinto Aluminium chief executive, said in a statement: "This is a very positive development for Rio Tinto Aluminum and an important step in our plans to develop new greenfield aluminium smelting capacity."
He said that the project could provide nearly 5,000 direct and indirect jobs, as well as help boost Malaysia's economy.
Goeneveld said: "CMS and Rio Tinto Aluminium are committed to develop the project to world-class environmental and community standards."
Syed Anwar Jamalullail, CMS chairman, has said that his firm's experience as Sarawak's leading infrastructure development company would complement technical expertise to be provided by Rio Tinto.
Jamalullail said that the project is seen as a "catalyst for the next phase of growth" in Sarawak, as the Malaysian government is developing as an industrial area sitting under vast energy resources of oil, natural gas and coal deposits.
Electricity for the smelter is expected to come from the Bakun hydroelectric dam.
The dam is currently under construction and has been strongly criticised because up to 10,000 tribal residents have been forced to relocate.
The $2.5bn dam project was approved in 1993.
Construction has involved the destruction of about 69,000 hectares of rainforests in Sarawak.
Much of its power output is to flow through undersea transmission lines in a programme to supply power to mainland Malaysia by 2010.
It has been delayed numerous times, and put on hold, but was revived two years ago.
Environmentalists have said that Rio Tinto's plan to tap power from Bakun is a move to cover up the dam project's failures.
Colin Nicholas, co-ordinator Sarawak's Centre for Orang Asli (Indigenous People), said: "From our point of view, by allowing the Rio Tinto project to go ahead, it is just like trying to cover up one natural disaster with another."
Nicholas said: "Another big project like this will see more green forestry being wiped out and that itself is another disaster."
Rio Tinto Aluminium is part of the Rio Tinto Group and owns mining, refining and smelting assets located mainly Australia and New Zealand.
The company is set to become the world's largest aluminium producer through its recent $38.bn takeover of Alcan, a Canadian aluminium company.