Often described as pragmatic, Mullen is the latest in a series of changes in the US military line-up since Gates replaced Donald Rumsfeld as defence secretary in December.
He is expected to address problems facing the US military, already at breaking point due to repeated deployments, as well as challenges beyond the Iraq war and the potential threat by Iran as an emerging nuclear power.
"The fighting in Iraq and Afghanistan will one day end. We must be ready for who and what comes after. That's the promise we've made"
Michael Mullen, new Joint Chiefs of Staff chairman
"The fighting in Iraq and Afghanistan will one day end. We must be ready for who and what comes after. That's the promise we've made," Mullen said on Monday.
"We owe it to the American people and to all of you - our men and women in uniform, and your families - to provide you with clear direction, outstanding equipment, and the focused policies you need to do your jobs."
Mullen, 60, who replaces outgoing chairman General Peter Pace, sailed through his confirmation hearings saying Iraq needed a political reconciliation and not a military solution.
"Barring that, no amount of troops and no amount of time will make much of a difference," he said.
Pace, whose six-year tenure spanned the wars in Afghanistan and Iraq, was denied the customary second two-year term as chairman after Gates concluded in June that his confirmation by a Democratic-controlled congress would be too divisive.
Observers say Mullen made his doubts about troop deployment in Iraq known to the Pentagon's civilian leadership before his nomination.
Army General David Petraeus, the US commander in Iraq, launched a new counter-insurgency strategy that has raised the number of soldiers deployed to their highest levels since 2003.
Chance for change
A military expert says Mullen could team up with Admiral William Fallon, the former Pacific commander who now heads US forces in the Middle East to decide to move away from Iraq.
Michael O'Hanlon, from the Brookings Institution, said Mullen and Fallon could be the ones to say that the US needs to scale back its involvement in Iraq "'because we are just straining the military and reducing our flexibility'".
However, Tom Wilkerson, a retired marine corps major general who is chief executive of the US Naval Institute, says Mullen has a "slim to none" chance of making major changes.
"He can react to outside events and give advice," he said. "But the idea that he would come in as the adviser to the secretary and the president to be able to make sweeping changes from where we're headed right now is probably a dream."