That position will now be filled by Steven Vanackere, the public affair minister.
Leterme's party won the last general election in June 2007 but he failed to form a coalition government until the following year.
He was forced out last December after it was alleged his aides had sought to influence a court ruling over the sale of Fortis bank after the financial crisis hit last year.
He was subsequently cleared of any wrongdoing however and was politically rehabilitated in July when Van Rompuy made him foreign minister.
"It's now his second chance. He has all the elements to prove he will be a good prime minister. I hope so for him and for Belgium," Van Rompuy said on Tuesday.
The Dutch-language newspaper Het Laatste Nieuws said Leterme's second chance would be his last, calling him Belgium's "most spoon-fed prime minister".
The reference was to the role of Jean-Luc Dehaene, another ex-prime minister, who has been given the job of drawing up proposals to resolve the country's institutional problems, a move by the king aimed at avoiding a repeat of the squabbling which beset Leterme's last term in power.
Belgium has struggled to reconcile the French-speaking Walloons with the country's Dutch-speaking Flemish population.
The Flemish majority in northern Belgium wants more power devolved to its own region, while the poorer Walloons in the south fear such a move would cause them to lose out politically and financially.
Flemish politicians have dominated recent governments, with Martens, Van Rompuy and Leterme all members of the Flemish Christian Democrats.
Leterme has also earned a reputation for gaffes. When asked by a reporter if he knew the Belgian national anthem, he stunned listeners by singing France's "La Marseillaise".
Other gaffes included saying that all the Belgian people share is "the king, the national football team and certain beers".
He disparaged French speakers in 2006 as lacking the "intellectual capacity" to learn Dutch and stoked controversy by calling his country an "accident of history" and saying it had no "intrinsic value" as a state.