The presidential office confirmed on Tuesday that Goh Kun had stepped down.
The PM had refused Roh's request to name nominees to three cabinet posts on Monday.
He said the request violated the spirit of the constitution and handed in his resignation.
Goh, who served as acting president for two months during Roh's impeachment trial, has been in the post since February 2003 when Roh began his term.
Roh is expected to nominate Goh's successor at the end of the week, according to a presidential Blue House official.
Parliament has to approve the nominated candidate.
Goh's widely anticipated move sets the stage not only for a new prime minister but also a likely cabinet reshuffle.