As nobody secured an absolute majority in the ballot of Socialist party members, the top two candidates go forward to the run-off vote on Friday.
Although she topped Thursday's poll, Royal will face an uphill task to maintain her lead, with critics accusing of her planning to shunt the party to the political centre.
A broad array of disparate party veterans have rushed to the side of Aubry, the daughter of Jacques Delors, the former European commission president.
Hamon, a left-wing member of the European parliament, also backed Aubry following the first ballot.
He said: "For our party to remain firmly anchored to the left, I ask those who supported me to vote massively for Aubry."
Francois Hollande, the current Socialist party leader and Royal's former partner, is stepping down after 11 years at the helm.
During his term as leader the party has lost two presidential elections, including Royal's heavy defeat against Sarkozy last year.
The winner of the run-off will be the favourite to stand as the socialist challenger to Sarkozy in the 2012 election.
Royal, 55, presents herself as a force for renewal and has promised to shake up the party if she wins, suggesting she will move its headquarters from a chic Paris neighbourhood to a more downtrodden area to get closer to ordinary people.
Speaking to supporters on Friday, Royal said: "Tonight's vote will allow us to start anew."
Aubry, 58, is mayor of the northern city of Lille and an former minister best known as the author of the 35-hour working week law, which Sarkozy has largely dismantled.
Some socialists have threatened to quit the party if Royal takes charge, with Jean-Luc Melenchon, a well-known legislator, already walking out, saying that support for Royal meant he no longer recognised the party as his political home.