Along with Li, the ministers of land resources and science and technology also stepped down on Friday, the official Xinhua news agency reported.
The change at the foreign ministry is unlikely to alter substantively China's foreign policy.
Major policy directions are set by the communist leadership, especially Hu Jintao, the Chinese president, who heads an interagency committee on foreign affairs.
Both the departing and new foreign ministers rose through the ministry ranks as American experts.
Both served as ambassadors to Washington at times of crisis in China's crucially important relations with the US.
But the two have markedly different styles.
Li was known for often striking an undiplomatic posture internationally.
As ambassador to the US, he refused to accept Washington's apology for what it said was the accidental bombing of the Chinese Embassy in Yugoslavia in 1999 during the Kosovo War.
In contrast the 57-year-old Yang has followed a more low-key approach.
Shortly after becoming ambassador to Washington, in 2001 Yang worked to defuse tensions between the two countries after a US spy plane collided with a Chinese fighter jet.
The US plane and its crew were held at a Hainan island air force for several days before eventually being allowed home.
A native of Shanghai, Yang studied at the London School of Economics in the early 1970s, as part of an effort to revive China's diplomatic corps devastated by persecution during the Cultural Revolution.
In the mid-1970s he served as an interpreter for George W Bush, the father of the current president, when he ran the US liaison office in Beijing.
The elder Bush reportedly gave Yang the nickname 'Tiger'.
However, during his time as ambassador to the US, Yang suffered a heart attack during his tenure and his health continues to be weak.
That has raised questions about whether he will be able to maintain a robust travel schedule as foreign minister.