Those on strike included police, healthcare workers, teachers and customs officials. Essential services are expected to be maintained through skeleton staffing.
"We have nothing to lose but the chains that bind us," Norman Mampane, the national spokesman for the Popcru police and prison guards' union, one of the groups in the labour coalition representing about 1.3 million state workers, said.
No marches were planned for Wednesday because the strike was announced so late the night before.
The strike was declared after four days of consultation in which the state's offer was to increase the monthly housing allowance to 700 rand from a previous offer of 630 rand.
However, it had refused to increase its wage rise offer of seven per cent.
"The strike is indefinite. It will go on until there has been an improvement, until the government delivers on our demands"
Fikile Majola, general secretary of the National Education Health and Allied Workers
A mid-range civil servant already makes about 40 per cent more in wages and benefits than the average South African worker, who takes home 6,383 rand ($880) a month.
Dumisani Nkwanba, the spokesman for the ministry of public service and administration, told Al Jazeera: "The government has been very sincere … The question here is the issue of affordability on the part of the state.
"The offer the government has put on the table … we feel is a fair and reasonable offer. We cannot move beyond what we have done now."
Al Jazeera's Haru Mutasa, reporting from Soweto, Johannesburg, said the unions do not believe the government when it says that it has no money.
"They say they see politicians living lavish lifestyles, they question why there was money for the football World Cup [staged in South Africa] and say they are tired of corruption allegations in government departments and that they will not put up with it anymore," she reported.
"They are saying that they are not going back to work until their demands are met and that they don't care how long the strike drags on.
"It's going to have a big impact. Schools are shut down … we've heard people have barricaded hospitals. If this strike drags on a lot of people will become the casualties of the strike action."
Anlysts expect a deal to be reached in the coming days, but economists fear that whatever deal is agreed will increase state spending as the government tries to bring its deficit down from 6.7 per cent of gross domestic product.
"Additional pressure on public sector finances and the subsequent need to increase the taxes will have consequential downstream impact on both business and consumers," the South African Chamber of Commerce and Industry said in a statement.
The last big public sector strike in South Africa took place in 2007 when a four-week strike by 600,000 state workers cost the economy several million lost man-days, discouraged investors and angered the public.