The case was filed on Wednesday, two weeks after Denmark's top prosecutor declined to press criminal charges, saying the drawings that sparked a firestorm in the Muslim world did not violate laws against racism or blasphemy.
Michael Christiani Havemann, a lawyer representing the Muslim groups, told The Associated Press the lawsuit sought 100,000 kroner ($16,100) in damages from Carsten Juste, Jyllands-Posten editor-in-chief, and Flemming Rose, culture editor, who supervised the cartoon project.
"We are seeking judgment for both the text and the drawings which were gratuitously defamatory and injurious," Havemann said.
He said the lawsuit was filed in the western city of Aarhus, where Jyllands-Posten is based. The court confirmed it had received the lawsuit but said a date for a hearing had not been set.
The newspaper published the 12 cartoons on September 30 with an accompanying text saying it was challenging a perceived self-censorship among artists afraid to offend Islam.
One of the drawings showed the Prophet Muhammad wearing a turban shaped as a bomb with a burning fuse. Another portrayed him holding a sword, his eyes covered by a black rectangle, and a third pictured a middle-aged prophet standing in the desert with a walking stick, in front of a donkey and a sunset.
The cartoons were seen as an insult to the prophet, depicting him as violent and primitive. Sunni Muslim tradition bans any image of the prophet, since depicting him risks insulting him or encouraging idolatry.
The newspaper apologised for offending Muslims after violent protests against Denmark erupted in the Middle East but stood by its decision to print the drawings, citing freedom of speech.
The cartoon row led to a boycott
of Danish products in the Gulf
Rose, who went on indefinite leave in February, and Juste could not be immediately reached to comment on the lawsuit Thursday. Jyllands-Posten spokesman Tage Clausen's only comment was: "We'll take it as it comes."
The petition, a copy of which was obtained by the AP, said the cartoons depict Muhammad "as belligerent, oppressing women, criminal, crazy and unintelligent, and a connection is made between the prophet and war and terror".
It said the drawings were published "solely to provoke and mock not only the Prophet Muhammad but also the Muslim population".