The emotional chanting crowds, mostly supporters of senior Shia leader Ayat Allah Ali al-Sistani, were let in on Friday morning by al-Mahdi Army fighters, who have resisted a siege by US-led forces for the past three weeks.
Later, many formerly besieged militiamen laid down their arms and melted away, and Iraqi police entered the mosque.
Representatives of al-Sistani said they reached an agreement last night with aides of al-Sadr to take over the highly revered religious site from the latter's militia.
"We have passed three-fourths of the way and we are at an agreement that retains the honour of all those involved," senior al-Sistani spokesman Hamid al-Khafaf told Aljazeera.
"There is full agreement among all the parties, and the doors will never be closed in the face of a settlement," he said.
The deal appears to pave the way for al-Sadr's movement to join the US-backed political establishment in Baghdad - an extraordinary turnaround for the Shia leader, who has been fighting US-led forces and demanding the removal of the government and all occupation troops.
"There is full agreement among all the parties"
Clashes in Najaf - marked by heavy US bombardment of the area around the Imam Ali mosque - and in neighbouring Kufa have left hundreds dead, including scores of civilians in recent weeks.
After an announcement that al-Sadr had ordered them to disarm, al-Mahdi Army fighters threw their guns into wooden carts around the mosque and slipped out of their black uniforms and into civilian clothing.
Some militiamen have disarmed
but others have kept their guns
But elsewhere in Najaf, many militiamen refused to give up their arms and reportedly carried them away to store in safe houses.
Interim Prime Minister Iyad Allawi had offered an amnesty to the fighters, coinciding with al-Sistani's dramatic arrival in the battle-ravaged city, on Thursday afternoon.
The Iraqi government said early on Friday al-Sadr would not be arrested - despite his armed challenge to its authority and outstanding warrants for his arrest.
The whereabouts of the Shia leader, who has been offered safe passage out of Najaf by Allawi, remain unclear and no direct word from him has been heard.
Al-Sistani reportedly persuaded al-Sadr late on Thursday to accept a deal ending the fighting in Najaf. The accord would demilitarise Najaf and neighbouring Kufa, with both US-led troops and al-Mahdi Army fighters withdrawing on Friday.
The revered Imam Ali mosque will be placed under the stewardship of Iraq's most senior Shia Muslim religious authorities, headed by Iranian-born cleric al-Sistani.
The mosque has been placed
under Shia authorities' control
Al-Sistani, who left for London to receive medical treatment just as the Najaf fighting erupted last month, returned to Iraq on Wednesday. Under British and Iraqi government escort, he led a convoy of followers into Najaf on Thursday to negotiate a peace agreement.
But the refusal so far of the al-Mahdi Army to disarm and disband across the country remains a potential obstacle.
If it succeeds, the accord will end a serious challenge to the authority of Allawi's fledgling government, which continues to battle resistance elsewhere in the country and is struggling to maintain vital oil exports.
'Not an army'
In an interview with Aljazeera, Shaikh Yusuf al-Nasri, head of public relations of al-Sadr's office in Baghdad said: "Al-Mahdi army must not be explained as an army.
"It includes honourable fighters from the Iraqi people who have been fighting to defend their country, land, sanctuaries and doctrine."
He added that the group comprised scientists, physicians, lawyers, workers and farmers.
"[Al-Mahdi Army's] arms are merely light and personal ones and there is no need to give attention to the disarming issue"
Shaikh Yusuf al-Nasri,
Head of public relations of al-Sadr's office in Baghdad
"This army does not have military camps or arsenals. It is ideologically known as al-Mahdi Army but in reality it is not an army like other armies," he said.
"There is nothing called arms. From the very beginning, al-Mahdi Army has waged a holy war against the occupiers. They do not possess weapons like those found in armies.They are the sons of Iraq. They fight by using their personal weapons."
Commenting on the latest accord, al-Nasri said: "The arms are merely light and personal ones and there is no need to give attention to the disarming issue."
But Dr Mazin al-Samarrai, a member of the Political Bureau of the Iraqi National Accord is not convinced of al-Nasri's description of the al-Mahdi Army.
"I have some doubts about Shaikh al-Nasri's assertion that al-Mahdi Army is composed of Iraqi national elements who seek the interests of Iraq. In any case, I would be honoured to be part of such movement," he told Aljazeera.