Khai, 72, has served as Vietnam's prime minister for two consecutive five-year terms - a decade of economic development and growing international relations for the once-isolated Southeast Asian country of 83 million people.
"I have asked to retire. We should let the younger generation lead," Khai said on Tuesday before the opening of the National Assembly, or parliament, which was expected to approve several leadership posts during its May 16 to June 29 session.
As for his successor, Khai said: "I have nominated my deputy, who has worked with me for eight years and was trained for this job."
Asked if it was Nguyen Tan Dung, the deputy prime minister, the prime minister said, it "should be him".
The retirement was expected because Khai was not re-elected to the politburo at April's Communist Party National Congress.
Dung, 56, is one of three deputy prime ministers in the one-party government system.
A career security officer and advocate of market reforms, Dung is generally considered as a representative of young leaders in Vietnam.
He was born in the southernmost province of Ca Mau.
Dung has taken a high profile in recent years and is often part of public events, such as February's announcement by the world's biggest chip maker, Intel Corporation, that it will build a $605 million assembly plant.
Last month, Dung had lunch with Microsoft founder Bill Gates when he visited Hanoi.
At the legislature on Tuesday, Dung greeted reporters with handshakes and smiles but declined to comment on changes in the leadership.
Khai visited the US in June
The assembly would receive Khai's retirement request and nomination of a new prime minister, the outgoing prime minister said.
In June, Khai became the first prime minister since the war and highest-ranking Vietnamese to make an official visit to Washington, where he met George Bush, the US president.
Bush will visit Hanoi in November for the annual summit of the 21-member Asia Pacific Economic Co-operation (APEC) forum, the biggest international event hosted by Vietnam.
"We want to be friends with all countries and we want to be their reliable partner," Khai said on Tuesday.
Last month, the five-yearly party National Congress chose general-secretary Nong Duc Manh, 65, to serve a second five-year term. The post is the highest-ranking political job in Vietnam.