The plaintiffs had wanted to prosecute Anders Fogh Rasmussen, the Danish prime minister, "as a representative of the state, for the illegal decision taken by the government and parliament on March 21, 2003, to enter into a war of aggression against Iraq".
They said that Denmark's involvement in Iraq violated article 19 of the country's constitution, which authorises war only in self defence or as part of a UN intervention.
Bjoern Elmquist, a lawyer for the plaintiffs, told the Danish Ritzau news agency that the group would appeal to the supreme court.
The court also noted that the Danish parliament's decision to authorise Denmark's military participation in the 2003 invasion had been renewed several times.
Denmark has 460 soldiers in Iraq, including 415 in the southern city of Basra under British command. Six Danish soldiers have been killed in Iraq since 2003.
Rasmussen announced in February that the Danish troops would be withdrawn in August, but that four helicopters and 50 crew would be sent to replace them.
Most of the plaintiffs were not directly involved in the invasion of Iraq and did not have family members stationed in Iraq, which weakened their case, legal commentators said before the ruling.