Donald Rumsfeld made the comments on Thursday during a question-and-answer session at the Harry S Truman Presidential Museum and Library in Independence, Missouri.
He paid tribute to former president Truman for countering Soviet expansionism with the Marshall Plan, Nato and a host of other international institutions.
"The United States did not win the Cold War by luck, and victory was not inevitable," Rumsfeld said.
"It took perseverance, it took a confidence in our force, despite the many uncertainties - and there were uncertainties throughout - and despite the many critics along the way who stood outside and blamed the United States," he said.
Rumsfeld acknowledged that the Cold War and the struggle against Muslim extremism are different in many respects.
"But we have to build partnership capabilities on the part of other nations, and make them better able to deal with terrorists inside their country"
US defence secretary
But he said that in both cases, the US had to prepare for a long, protracted struggle, punctuated by periods of military conflict.
Rumsfeld also said no country could deal with extremism on its own.
"We can put pressure on them and we are," he said, referring to Muslim extremists.
"But we have to build partnership capabilities on the part of other nations, and make them better able to deal with terrorists inside their country."
Call for partnership
He cited that as another similarity with the Cold War - "the critical importance of being able to bolster the capacities of partner nations".
"This notion was the heart of the Marshall Plan, which cost more than $100 billion in today's dollars, but most certainly helped to save Western Europe from Soviet tyranny and led to the emergence of important democratic allies that, despite our occasional differences, remain indispensable to our success today," he said.
With US military forces stretched thin, a major Pentagon strategy review last month highlighted, among other things, the importance of alliances and partnerships.
US relations with key European allies were badly strained by the US-led invasion of Iraq in 2003 and Rumsfeld himself has often been at the centre of diplomatic rows.