The protesters demanded that he step down immediately and call general elections.
Bajnai, 41, who had been serving as the economy minister, is a member of the Hungarian Socialist Party.
He replaces Ferenc Gyurcsany who announced his decision to resign in March after criticism of his handling of the country's economic crisis.
Hungary has been badly hit by the global financial downturn and last year needed a $25bn IMF-led rescue package to prevent the collapse of the economy.
In one street in the city, demonstrators were reported to have set fire to a European Union flag, before spitting and urinating on it.
Police officers said nine protesters had been taken into custody.
Another group of demonstrators were said to have headed for the office of Laszlo Solyom, the Hungarian president, to demand the dissolution of parliament and early elections.
Police were blocking access to the president's office.
Earlier in the day, protesters had set several geese free in the city centre, as a reference to the new prime minister's failure to save a poultry-producing plant in 2003.