Manasseh Sogavare, who was prime minister between 2000 and 2001, defeated Fred Fono by 28 votes to 22 in a secret parliamentary ballot on Thursday, a government spokesman said.
Sogavare came in third behind Rini and Job Tausinga in national elections in April, but his supporters backed Rini in the second round, helping Rini to overtake Tausinga to land the post of prime minister.
Sogavare was named commercial minister in Rini's short-lived cabinet but, after being promised the premiership if he defected, was one of several ministers who crossed over to the opposition camp, precipitating Rini's resignation.
His election was greeted by a cheering crowd of around 300 people outside parliament in the South Pacific nation's capital, Honiara, amid tight security by foreign troops and police.
The newly elected prime minister urged the crowd to celebrate his election peacefully.
"The parliament has made a decision for change. This is an important change that people in the Solomons have been looking forward to," he said.
Sogavare won in a secret
Nearly 400 troops and 120 police officers from Australia, New Zealand, Fiji, Tonga and Papua New Guinea were sent to Honiara to help quell protests over Rini's election on April 18, which spiralled into two days of violent rioting and looting.
The tiny Chinese business population in Honiara was targeted and most of the city's Chinatown was destroyed because of rumours that aid money from Taiwan was used to help elect Rini and that his government was heavily influenced by local Chinese businessmen.
Two politicians opposed to Rini have been charged in relation to the riots and remanded in custody, but both were allowed to vote from their cells in the latest leadership ballot.
Voters ousted half the parliament in the election, which was dominated by corruption. Seven ministers in the last government faced graft charges.
The Solomons, a nation of 992 islands spread over 1.35 million square km and once a British protectorate known as "The Happy Isles", was on the brink of collapse in 2003 because of ethnic fighting, prompting Australia to lead a multinational peacekeeping force to restore peace.