The association said US troops burst into the Umm al-Qura Mosque complex in western Baghdad at 3am (0000 GMT) on Sunday, blowing doors off hinges and ransacking offices.
A source in the office of Shaikh Harith al-Dhari, who heads the Association of Muslim Scholars (AMS), said: "They took Shaikh Unis al-Ugaidi [a member of the association], two employees and two guards.
"We do not know the reason, but it's obvious, it's due to the association's attitude towards the occupation."
He said US troops had disarmed about 20 bodyguards stationed in the complex and confiscated their weapons.
The AMS is an influential group of Sunni Muslim scholars who hold sway over many Muslims, especially in the western governorate of Anbar, heartland of the revolt against the US presence in Iraq.
Its leaders have called on US forces to withdraw from Iraq and boycotted the 15 December parliamentary election.
Reuters Television footage showed spent shotgun shells and explosive charges lying on the ground of the offices. Many office doors showed signs of forced entry.
A helicopter crashed in Iraq on
Sunday, killing 12 Americans
In one room, cupboards used to store the shoes of worshippers had what appeared to be Christian crosses scrawled on them.
Lieutenant-Colonel Barry Johnson, a US military spokesman, said the raid had been deliberately timed to minimise the risk to civilians and that the soldiers had respected the fact they were in a place of worship.
He rejected as "unbelievable" any suggestion they may have been responsible for the crosses.
In a related development, US and Iraqi forces raided on Sunday villages in Abu Ghraib area west of Baghdad. The villages are the home of the Zowba tribe to which al-Dari belongs.
The AMS issued a statement condeming the raid, and claimed that 35 people have been arrested including women, children and elderly men.
The statement called on called on "occupation forces" and the Iraqi National Guard immdiately to release six women who it said were arrested in Sunday's raid.