Relations between Denmark and the Muslim world came under strain in 2005, during Rasmussen's premiership, after Danish newspaper Jyllands Posten printed cartoons of the Prophet Mohammed.
Despite global protests, Rasmussen refused to condemn the publication.
This year, in a speech in the Turkish city of Istanbul, Rasmussen called for a balance between free speech and a respect for religion.
Afghanistan, where US-led forces are battling the Taliban, will be one of Rasmussen's biggest tests.
"There is certainly a hard core [Taliban] that is impossible to reach any deals with. They have only respect for military powers"
Anders Fog Rasmussen,
Nato took over some operations in the country six years ago.
But a steadily rising number of civilian casualties because of Nato attacks has fuelled Afghan anger against western forces.
Haroun Mir, an Afghan political analyst, warned that Nato's new secretary-general had a long way to go.
"The mission of the new chief of Nato will be very difficult," he told Al Jazeera.
"Firstly to convince some of these countries to be more active in fight against terror in Afghanistan, secondly to increase their number of troops, and thirdly to prevent some of these countries who might pull back their forces from Afghanistan."
In an interview published in the Danish newspaper Politiken on Saturday, Rasmussen said he would support dialogue with "moderate groups on the outer reaches of the Taliban".
"There is certainly a hard core that is impossible to reach any deals with. They have only respect for military powers," he was quoted as saying.
"But there are groups that you can talk with to try and bring about some kind of reconciliation with the Afghan community."
He continued: "I certainly don't know why one should strike any agreements with those who are killing our soldiers. We are talking about other groups located on the outer fringes of the Taliban."
Other European officials, including David Miliband, the Britain's foreign minister, have struck similarly conciliatory notes about engaging Taliban fighters willing to renounce violence.