The protesters shanted slogans and demanded the resignation of Emil Boc, the prime minister.
"We will not leave until the government quits," Bogdan Hossu, leader of the Cartel Alfa trade union, said.
He warned unions would call a general strike on May 31 if their demands were not met.
The importance of the IMF deal has been highlighted by two failed debt auctions this month as many investors fear the government will bow to public pressure and not follow through with the cuts.
The IMF has said it will disburse its next tranche of aid only after Romania enforces a credible plan to reduce its budget deficit to 6.8 per cent.
It was 7.2 per cent in 2009 and the IMF says that without cuts it could reach 9 per cent of GDP.
"The government's ability to pass those cuts is being tested and remains particularly important given the suspension of IMF disbursements," BNP Paribas, a French bank, said in a statement.
Romania's state sector, criticised for inefficiency and corruption, employs one-third of the country's workforce.
Pensions, wages and other social benefits account for two-thirds of budget revenues.
Such spending, including triple-digit pay rises in the booming 2005 to 2009 period, has become increasingly hard to finance with the private sector facing shrinking domestic demand and rising unemployment.