Van Rompuy's appointment follows a six-day mediation mission between the various members of the coalition by Wilfried Martens, a former prime minister.
Belgian media said Martens' biggest task had been persuading a reluctant Van Rompuy, who has the reputation of being both an intellectual and a budgetary hardliner, to accept the job.
Van Rompuy, 61, was budget minister between 1993 and 1999, bringing Belgium's debt down sharply from some 130 per cent of gross domestic product in his first year in office.
He has expressed doubts about the current government's plans to spend its way out of the current economic downturn.
Carl Devos, a political scientist at the University of Ghent, said: "It is a bit of a surprise as he was refusing to do it. I think the pressure on Van Rompuy was huge.
"He has respect from all parties. He's very experienced, but he's never led a government and this is certainly not the easiest time to be taking over."
Fortis investors, angry at seeing their share price plummet, won an appeal court ruling earlier this month freezing the proposed sale of most of the group's assets to BNP Paribas, a French bank.
They argue the deal should be renegotiated.
Van Rompuy referred to himself in an interview in Belgian newspaper De Standaard newspaper on Saturday as being in the "autumn" of his political career.
"I feel myself to be anything but indispensable," Van Rompuy told De Standaard.
He appears to have been the only figure acceptable to all members of the five-party coalition, being respected both in the Dutch- and French-speaking parts of the linguistically divided nation.
An ability to bridge the cultural divide will be important if Belgium is to avoid lurching back into crisis and prompting speculation about the 178-year-old country breaking in two.
Devos said it was unclear how long Van Rompuy would be in charge.
It could be until 2011, when the next federal elections are due to be held, but it could just a matter of months, when Leterme might have been cleared by a parliamentary commission.