"I am not going to take this," Royal told the AFP news agency after Aubry's supporters claimed victory in the early hours of Saturday.
Royal convened a crisis meeting of her team and Jean-Pierre Mignard, her lawyer, said she was seeking a re-vote for next week.
"We have noted that there were many disputes that arose here and there concerning the voting operations," Mignard said.
"These irregularities should lead, at the very least, to multiple verifications that may be difficult and even impossible to make.
"That is why Segolene Royal is proposing that we vote again, which appears to be the only acceptable and dignified solution."
"We have noted that there were many disputes that arose here and there concerning the voting operations"
Jean-Pierre Mignard, lawyer for defeated ex-presidential candidate Segolene Royal
Aubry, the 58-year-old mayor of Lille, rejected the proposal and said "there is no reason" for a new vote.
Francois Hollande, the outgoing leader, called an emergency meeting of the party's top council.
Aubry's victory capped weeks of bitter campaigning that deepened divisions within the party, with Aubry warning last week that Socialists needed to "get their act together" or face extinction.
Royal had come out ahead in the first round and hoped to lead the party's renewal and stand again as its candidate for president in the 2012 vote.
The 55-year-old president of the Poitou-Charentes regional council had promised to reshape France's left by opening the party up to a younger membership and possibly forging an alliance with centrists to beat Sarkozy.Order of business
Aubry believes that the party should stay to the left, and that a shift to the centre would alienate its traditional voter base at a time when the financial crisis has revived state-driven economics.
Aubry's first job will be to unite the party and show that she is not the captain of a sinking ship.
As labour minister in the late 1990s, Aubry drafted legislation creating the 35-hour work week, a Socialist measure that Sarkozy has sought to unravel and which has been criticised even within the party.
After three consecutive defeats in presidential elections, the Socialists have been mired in internal differences and unable to compete effectively with Sarkozy since he took office last year.
Battling nearly a year of low approval ratings, Sarkozy has recently seen his figures bounce back over his hands-on response to the financial crisis and France's high-profile presidency of the European Union.