''I've come to Mosul today to follow up on our victories and on the eradication of ISIL, which we are close to achieving," Abadi said on Sunday.
Abadi's office had said he was visiting "liberated" Mosul to congratulate his "heroic fighters", but the prime minister later indicated he would only declare victory once final pockets of resistance were cleared.
"Most ISIL fighters in Mosul have been killed and just a few are left now, he said. "I will leave some room for the heroic armed forces to complete this action so we can announce victory soon, God willing."
Photographs released by his office showed Abadi dressed in a black military uniform and cap, shaking hands with police and army officers as he arrived in the city.
The prime minister joined hundreds of civilians in Mosul to celebrate the army's gains over ISIL.
Convoys of cars were seen driving through the Old City, as people chanted victorious slogans and waved national flags.
The operation in Mosul has been backed by a US-led coalition battling ISIL in Syria and Iraq, which has carried out waves of air strikes against the group and deployed military advisers on the ground.
Emmanuel Macron, the president of France, whose country is a key part of the coalition, was among the first world leaders to offer his congratulations.
"Mosul liberated from Daesh," he tweeted, using an Arabic acronym for ISIL. "Homage from France to all those, with our troops, who contributed to this victory."
Abadi arrived in Mosul after state TV reported that troops had reached the Tigris riverside and raised the Iraqi flag there.
'Far from over'
Despite the celebrations, heavy fighting by holdouts continued in parts of the Old City neighbourhood near the Tigris River.
Feisal Istrabadi, former Iraqi ambassador to the UN, told Al Jazeera that the fight against ISIL in Iraq is far from over.
"We've already seen ISIL engaging in a campaign of terror in Baghdad and other places, so I think its immediate future is that it runs as a classical terrorist organisation pulling off terrorism acts until the state of Iraq is able to adequately deal with it or snuff it out altogether," he said
About 100 ISIL fighters had earlier been reported to be trapped in a sliver of the Old City along the Tigris.
Many of them threw themselves into the river earlier on Sunday as they faced imminent defeat, according to Reuters news agency.
The agency also reported plumes of smoke over the Old City and decaying corpses of ISIL fighters lying on its streets.
Scattered bursts of gunfire could be heard and several air strikes were carried out, it said.
The fall of Mosul would be the biggest defeat yet for ISIL three years after it seized the city in a lightning offensive.
With air support from the US-led coalition, Iraqi forces launched the battle for Mosul in October, retaking the eastern part of the city in January and starting the operation for its western part the next month.
The nearly nine-month battle for Mosul has ruined parts of the city, killed thousands of civilians and displaced nearly one million people.
While the armed group has lost much of the territory it once held in Iraq, Al Jazeera's Charles Stratford, reporting from Erbil in northern Iraq, said ISIL still controls a number of towns in Iraq, including Tal Afar, west of Mosul, and Al Qa'im on the Syria-Iraq border.
"By no means does this seemingly imminent victory by Iraqi forces in Mosul mean the end of ISIL in Iraq," he said. "There's still a lot more to do."
Source: Al Jazeera and news agencies