Fillon said he accepted his share of the blame for the defeat, and said the government would press ahead with reforms that include an overhaul of the pensions system, and would make jobs and economic growth an "absolute priority".
Sunday's elections were regarded as a major test for Sarkozy and an informal kick-off to the 2012 presidential race.
Martine Aubry, the Socialist party leader, called on the left to retake power at a national level.
"The French have expressed their rejection of the politics of the president and the government," she said.
Sarkozy was elected in 2007 on promises to boost France's economy and get people back to work, but last year's recession has driven unemployment up to 10 per cent, its highest level in a decade.
Fears about job losses, immigration and security, as well as resentment about issues such as bank bail-outs and executive pay have all dented Sarkozy's popularity, which has fallen since he won approval for his handling of the financial crisis in 2008.
The powerful CGT union has called for a day of protest over wages and pensions on Tuesday.
Other groups, such as farmers, may also see a big defeat for the centre-right as an opportunity to put pressure on the government.
There has also been increasingly open criticism of the leadership from within the UMP, where some well-known party figures including Alain Juppe, the former prime minister, are calling for a change of course.
Jean-Francois Cope, the head of the UMP parliamentary group and one of Sarkozy's main rivals in the party, said: "We can't hide from this.
"This is obviously a real defeat for us. You can't beat about the bush. That's the reality of it."
The centre-right suffered one of its worst losses in years in the first round of the ballot on March 14, with Sarkozy's party scoring just 26 per cent as the Socialists took 29 per cent and the broader left parties combined claimed some 50 per cent.
Regional councils in 25 regions, 22 on the French mainland and three overseas territories, were up for grabs, the Caribbean island of Guadeloupe having already been won outright by the Socialists in the first round of voting.