The Islamic Supreme Council of Canada, based in Calgary, said on Monday it filed a complaint against the Western Standard, a right-wing weekly newsmagazine that is reprinting eight of the 12 cartoons.
The cartoons have sparked protests around the world and riots in some Muslim countries.
Syed Soharwardy, president of the council, said his group has not seen the magazine's latest edition but would try to have Calgary police investigate the Western Standard for hate crimes.
"This is a provocation of Canadians and an insult to our religion," he said.
"We are convinced that within the boundaries of Canada there is a way that we can prosecute these people. The whole Muslim community is seriously looking into this and consulting with lawyers."
Grounds for prosecution
Canadian law prohibits the wilful incitement of hatred towards any identifiable group, based on ethnicity, region or sexual orientation.
While the cartoons have been reprinted by several European newspapers, no major Canadian newspaper has run the series.
One of the cartoons was reprinted by Le Devoir in Montreal. A University of Prince Edward Island student newspaper printed the cartoons but copies were ordered off the campus.
"This is a provocation of Canadians and an insult to our religion"
Islamic Supreme Council of Canada
The Globe and Mail, published in Toronto, said on Saturday that running the cartoons "would be both gratuitous and unnecessarily provocative".
Ezra Levant, publisher of the Western Standard, told CBC television the magazine decided to publish the cartoons because it considered them newsworthy.
"We're not publishing them for their editorial merits," he said.
"They're boring cartoons. They're bland. They're not interesting. We're not running them because we share their views. We're running them because they're the central fact that caused Muslim radicals around the world to riot."
Meanwhile, Kofi Annan, the UN secretary-general, has said Iran, Syria and other governments that failed to protect foreign embassies from mobs protesting over the cartoons should pay for the damage.
"The government has a responsibility to prevent these things from happening. They should have stopped it, not just in Syria or Iran but all around," Annan said.
In Lebanon, protesters torched
the Danish consulate
"Not having stopped it, I hope they will pick up the bill for the destruction," he told CNN. "They should be prepared to pay for the damage done to Danish, Norwegian and the other embassies concerned."
Danish facilities have been singled out for attacks, including diplomatic missions in Syria, Lebanon and Iran.
Iranian Holocaust competition
In Iran, a Brazilian has become the first person to enter a newspaper contest for cartoons on the Holocaust.
The contest was devised by Hamshahri, one of Iran's five leading newspapers. The newspaper said its contest was a test of the Western world's readiness to print cartoons about the Nazi slaughter of six million Jews in the second world war.
"We don't intend retaliation over the drawings of the prophet. We just want to show that freedom is restricted in the West"
The first entry depicts a man, smoking a cigarette and wearing a prison uniform, with a wall and guard tower in the background.
The man, with a moustache and several days growth of beard, is wearing a white keffiyeh and has his right hand over his forehead and eyes.
On his chest is a red crescent with a letter "P". Below is the number 7256, the significance of which was not immediately clear, although Israel is said to be holding about 8000 Palestinian prisoners.
Davood Kazemi, executive manager of the contest and cartoon editor at the paper, said: "We don't intend retaliation over the drawings of the prophet. We just want to show that freedom is restricted in the West."