Opponents of the plan, to build a $100m Islamic centre two blocks from the so-called Ground Zero site where the September 11, 2001 attacks took place, appeared to outnumber supporters.
Singer Bruce Springsteen's "Born in the USA" blared over loudspeakers as mosque opponents chanted, "No mosque, no way!"
They say the proposed location is insensitive and fear it will harbour religious extremism.
One sign read: "Not all Muslims are terrorists, but all the terrorists were Muslim."
Meanwhile, those who back the centre cited the right to religious freedom and the need to promote tolerance and understanding.
John Terret reports from rallies held in New York over proposed Islamic cultural centre
"We don't care what bigots say, religious freedom is here to stay," they chanted in the demonstration.
Ilene Kahn, a retired school teacher, said: "This has become a political tool to preach hatred. The peace-loving Muslims did not attack us."
The dispute has sparked a national debate on religious freedom and American values and is becoming an issue on the campaign trail ahead of the mid-term congressional elections.
Barack Obama, the US president, and Michael Bloomberg, the New York mayor, have earlier said they support the right of Muslims to build the centre near Ground Zero, while Republicans, including former vice presidential candidate Sarah Palin, oppose it.
Others suggest it be moved to a less controversial spot.
Some opponents have taken legal action, seeking to void a ruling that would allow construction to proceed, while some construction workers have launched a Hard Hat Pledge, vowing not to work on the project.
The Cordoba Initiative, the group behind the project, describes it as a Muslim-themed community centre with a view of making it a hub for interfaith interaction, as well as a place for Muslims to bridge some of their faith's own divisions.
Plans for the project include a 13-story building to house an auditorium, swimming pool, meeting rooms as well as the prayer space. The structure is architecturally plain and does not include a minaret, dome or other motifs often associated with mosques.