New York Mayor Michael Bloomberg said on Wednesday the gunman who fired the shots in the main chamber of the Hall is still at large.
However, an hour later there were more unconfirmed reports that the gunman had been shot dead by armed police called to the scene.
Bloomberg who has his offices in the chamber was present at the time of the shooting but was unhurt.
A police spokesman said the incident occurred just after 1800 GMT as council members were gathering for a session.
Councilman Michael McMahan said the gunman, who he described as male, in his late 40s and wearing a khaki suit, had been in the public balcony overlooking the chamber.
"I saw two men walking on the balcony and then one stopped, and all of a sudden there was a gun and then he fired shots into a person in front of him," McMahan was quoted as aying.
There was a brief pause and another shot and it appeared that the shooter was struck," McMahan said. "Then all pandemonium broke out."
"It is not terrorism. It appears to be a random act but we cannot allow this to go on," Bloomberg said. "This is an attack on democracy."
Police had cordoned off the City Hall building and all council members had been taken into a commitee room where they were being guarded by uniformed officers, officials said.