An interior ministry source said that 20 employees from the higher education ministry were seized, but a spokeswoman for the department itself said "100 or maybe 150" had been taken, including visitors to the building.
The gunmen drove up to the ministry's research directorate in the religiously mixed Karrada district in government vehicles.
Brigadier Abd al-Karim Khalaf, the interior ministry spokesman, said: "All interior ministry forces are on alert, searching for this group. We don't know if it's terrorists, militias or even government forces."
Alaa Makki, the head of the parliamentary education committee, said both Shias and Sunnis had been abducted in the raid and urged the government to rapidly respond to what he called a "national catastrophe".
Makki said the gunmen had a list of names of those to be taken and claimed to be on a mission from the government's anti-corruption body.
A female professor visiting at the time of the kidnappings said the gunmen forced men and women into separate rooms, handcuffed the men, and loaded them aboard about six pickup trucks.
"All interior ministry forces are on alert, searching for this group. We don't know if it's terrorists, militias or even government forces"
Brigadier Abd al-Karim Khalaf, the Iraqi interior ministry spokesman
The minister of higher education said that teaching in all of Baghdad's universities would be halted after the mass abduction.
Abed Dhiab al-Ujaili said: "We have no other option than to halt teaching in universities, at least in Baghdad, until we find out what happened.
"We are not ready to lose more professors."
Academics are increasingly being singled out in sectarian violence, and thousands of professors and researchers have fled from the country.
A university dean and a Sunni geologist have been murdered in recent weeks. At least 155 education workers have been killed since the war began.
The security forces have been accused of taking part in, or turning a blind eye to several previous mass kidnappings which are believed to have been carried out by sectarian groups.
The Sunni minority have blamed many of the kidnappings on armed groups from the now dominant Shia parties, who control the interior ministry.
The higher education ministry is headed by a member of the main Sunni Arab political bloc.
In Shula, a predominantly Shia area of western Baghdad, mourners protested about what Iraqi officials said was a US raid that killed six people.
The US military declined to confirm any operation in the neighbourhood.
Interior ministry sources said 13 people were also wounded after US troops called in an air strike when they came under fire from al-Mahdi Army fighters loyal to cleric Muqtada al-Sadr.
Angry mourners chanted slogans criticising the US and supporting al-Sadr.