The gunmen, who wore camouflaged Iraqi police uniforms, used at least ten vehicles for the kidnapping on Saturday, police officials said.
They entered the room where a meeting was taking place and asked everyone to stand facing the wall, sources said.
The building’s guards were not taken, but the kidnappers blindfolded and handcuffed everyone in the room.
Mahmoud said Ahmed al-Hijiya, president of the committee, was taken at around 1:30 pm (0930 GMT) along with other employees as they attended a conference in Karradah, a Shia neighborhood in Baghdad.
Others seized included the deputy head of the Olympic committee, Ammar Jabbar al-Saadi, the chairman of the Taekwondo Federation, Jamal Abdul-Karim, and the chief of the Boxing Federation Union, Bashar Mustafa.
The director of sports medicine, Dr Faleh Francis, was originally reported to be among those kidnapped, but state-run television later said he was not part of the group.
It is unclear how many people were abducted in all.
Security guards outside the meeting did not interfere because they thought the kidnappers were legitimate law enforcement officers, police said.
Two guards were killed; one while trying to flee the building, the other was shot dead and dumped on a street in Karradah.
The kidnappings come a day after Iraq’s national wrestling team pulled out of a tournament in the United Arab Emirates when its coach was killed in Baghdad.
Iraq’s Olympic Committee was dominated by Saddam Hussein’s son until the US-led invasion of 2003.