Iraq has asked NATO to put pressure on Turkey to withdraw its troops from northern Iraq immediately.
The request came after Turkey said it would not deploy any more soldiers but refused to pull out those already there.
The arrival of a heavily armed Turkish contingent near the front line close to Mosul is part of a mission to train and equip Iraqi forces in the fight against the Islamic State of Iraq and the Levant (ISIL) group, according to Turkey.
However, the Iraqi government says it never invited such a force.
"NATO must use its authority to urge Turkey to withdraw immediately from Iraqi territory," Haider al-Abadi, Iraqi prime minister, said in a statement posted on Tuesday after a 48-hour deadline set by his government for a withdrawal of the troops expired.
Abadi spoke to Jens Stoltenberg, NATO secretary-general, by telephone, the statement said, calling the deployment a violation of Iraq's sovereignty.
Russia, already angry with Turkey after the Turkish air force shot down one of its jets flying close to the Syria border last month, said it considered the force's presence in Iraq illegal.
However, Iraq's ambassador to the UN appeared to play down the dispute on Tuesday, saying bilateral talks between the neighbouring states to end the row were proceeding favourably.
"We're solving it between Baghdad and Ankara bilaterally," Mohamed Ali Alhakim, Iraqi ambassador, said after Russia raised the issue of Turkey's deployment during a closed-door meeting of the Security Council.
OPINION: Playing politics with boots on the ground
"We have not yet escalated it to the Security Council or to the United Nations. For us, what is helpful is the bilateral discussion going on right now between Baghdad and Ankara, and it's going extremely well."
Alhakim said Russia had not consulted Iraq before raising the issue.
Russia's increasing involvement in the Syrian conflict has put it at odds with Turkey, which has strongly supported rebels fighting President Bashar al-Assad.
Vitaly Churkin, Russia's UN ambassador, suggested he was disappointed that the 15-member Security Council did not issue a statement reaffirming the principles of territorial integrity and national sovereignty.
"We believe that Turkey has acted recklessly and inexplicably, carrying out additional deployments on the territory of Iraq without the consent of the Iraqi government," Churkin said.
Meanwhile, Masoud Barzani, the Kurdistan Regional Government president, is expected to hold talks with Turkish officials, including Ahmet Davutoglu, Turkish prime minister, in Ankara on Wednesday.
Earlier Davutoglu said he wanted to visit Baghdad as soon as possible to try to calm the row.
"Training at this camp began with the knowledge of the Iraqi defence ministry and police," Davutoglu told a meeting of deputies from his ruling AK party.
"Those who make different interpretations of the Turkish military presence in Mosul are involved in deliberate provocation."
The Pentagon has urged both countries to resolve the issue diplomatically, and noted the Turkish deployment was not part of a US-led coalition battling ISIL.
Source: Al Jazeera and agencies