Amid criticism that he had not shown sufficient concern for the troops' welfare during his last visit in December, Rumsfeld said on Friday he would be thanking army personnel and wishing them merry Christmas.
After landing he visited staff and patients at the 67th Combat Surgical Hospital that dealt with the Mosul bombing, a deadly attack that killed 22 people in a military canteen.
However, Rumsfeld said the trip had been planned for a while but was kept secret for security reasons and he said the attacks had not been a factor in selecting Mosul to visit.
His trip comes after a number of calls for Rumsfeld's resignation from Democrats in the US and even some Republicans who are unhappy with his performance.
Rumsfeld was criticised for appearing insensitive after soldiers told him they scavenged scrap metal to protect their vehicles and for the disclosure that a machine had signed his letters to the families of soldiers who died in Iraq.
An uncharacteristically subdued defence secretary responded on Wednesday and said the grief of the bereaved families was "something I feel to my core".
Rumsfeld will also take breakfast with troops in Mosul and be briefed by commanders on the area's security situation which was deteriorating badly even before Tuesday's attack.
"I don't doubt for a moment that some of these folks in Falluja went up to Mosul," he said.
The defence secretary later travelled south to another army base just outside Falluja.
He acknowledged to troops that the army had suffered "setbacks" in recent weeks.
However, Rumsfeld added: "All along the way, it is bumpy and it's tough. It is not a smooth, easy path to success, there are setbacks but you will look back when you are about my age and you will be proud."