The elections were the first electoral test of Sarkozy's popularity since his victory last May, but the mood has changed sharply since then as the economy has stumbled.
Government leaders maintained that voters had cast their ballots based on local issues and that the result would not affect national policies.
Jean-Francois Cope, leader of the UMP parliamentary party, said: "This is an evening of defeat."
Socialist leaders immediately called on the government to listen to voters and abandon "unjust" reforms ,but Francois Fillon, the prime minister, said there would be no change of direction.
'The bling-bling president'
Voter disapproval of Sarkozy's lifestyle and personal style combined with fears over the cost of living and eroding spending power have sent his personal approval ratings plummeting.
Aides have suggested an image makeover was needed to rekindle voter approval of the 53-year-old president, criticised for a brash and at times extravagant style that earned him the nickname "the bling-bling president".
Fewer than four in 10 voters now approve of his performance. Last July his ratings stood at 67 per cent.
Xavier Darcos, the education minister, lost his position as mayor of Perigueux in the south, but there was no indication he would lose his cabinet seat.
Francois Bayrou, head of the centrist MoDem party, who came third in last year's presidential election, was beaten in the race for Pau in the south, but pledged to continue with his drive to build a major third force in French politics.
"There will be other battles, there will be other fights and I promise you, there will be other victories," he said.