Rumsfeld arrived at the US base in the southern city of Kandahar on Wednesday, US military spokeswoman Lieutenant Cindy Moore said.
Moore provided no details of his schedule, although an Afghan government spokesman said Rumsfeld also was expected in the capital, Kabul, for talks with Karzai.
Officials in neighbouring Pakistan said Rumsfeld was expected in Islamabad later on Wednesday for meetings with the president, General Pervez Musharraf, and other leaders.
The visit comes as US and Afghan officials discuss a "strategic partnership", possibly including permanent base for the US military, which continues to broaden its role in Afghanistan.
US commanders recently agreed to provide intelligence on targets for a crackdown on the country's illegal narcotics industry, the world's largest, and to extend training of the new Afghan army to include the police.
Rumsfeld may press Kabul and Islamabad - key allies in Washington's "war on terrorism" - to improve relations clouded by the ability of Taliban-led fighters to launch attacks in Afghanistan from safe havens in Pakistan.
US and Afghan officials are
eyeing a 'strategic partnership'
US officials repeatedly have called on Karzai to announce a reconciliation programme to encourage former Taliban to lay down their arms and support the government in return for freedom from prosecution.
But Karzai has resisted, apparently because of opposition within his government and his wish to extend the programme to other parties in the country's long civil wars.
Officials say a few dozen former Taliban members have come forward, but none appears to have much influence over the rebels still fighting the 17,000-strong US force in Afghanistan and its Afghan allies.