Gates said he had intended to name Pace to a second two-year term as chairman in September, but changed his mind after consulting members of congress.
"I have decided that at this moment in our history, the nation, our men and women in uniform and General Pace himself would not be well served by a divisive ordeal in selecting the next chairman of the Joint Chiefs of Staff," he told reporters.
The chairman of the Joint Chiefs of Staff is the president's chief military adviser and the most senior US military officer.
Gates has chosen Admiral Michael Mullen, who is currently the chief of naval operations, to replace Pace.
Gates also had to replace Admiral Edmund Giambastiani as vice-chairman so that the two most senior military positions would not both be held by naval officers.
Giambastiani will be replaced by Marine General James Cartwright, currently the head of the US Strategic Command, which is responsible for US strategic nuclear forces.
Pace will step down in September after a 40 year career in the marines that began as the leader of a rifle platoon in Vietnam in 1968.
He commanded US troops in Latin America, served as director of the Joint Staff, and was deputy commander of US forces in Somalia during an ill-fated intervention in the early 1990s.
|Pace has been at the centre of military |
decision-making for six years[AP]
He was named vice-chairman of the Joint Chiefs of Staff weeks after the September 11, 2001 attacks, becoming the first marine in history to hold the post.
Pace quickly became a favourite of Donald Rumsfeld, the former US defence secretary, who raised his profile by sharing the podium with him at press conferences.
Mullen, a surface warfare commander who studied at Harvard Business School, served as commander of US naval forces in Europe and was in charge of Nato operations in the Balkans, Iraq and the Mediterannean.
He became the chief of naval operations in July, 2005.
Gates called him "a very smart strategic thinker, and I think he has a view of the interests of the services as a whole".