The 30m ferry, which has a capacity of 150 passengers and can travel at up to 22 knots, was returning to dock and was not carrying passengers. It suffered only minor damage.
Judy Bosler, president of the Australian Professional Skaters Association, said Alan Blinn and Simone Moore, two international-level skating judges were killed, along with an unidentified man and that the country's skating community was devastated.
Morgan Innes, a teenage skater, was still missing on Thursday. Liz Cain, a former Australian Olympic figure skater, was badly injured in the incident, losing part of one of her legs.
Bosler said: "It's an absolute tragedy for us because it's a small community and we're all reliant on each other."
A witness on a passing ferry told local radio he heard a thud and then an announcement that the boat he was on was going to the aid of a vessel in distress.
Clive Marshall, on ABC Radio, said: "There were people in the water, there was a lot of wreckage, lots of shouting, mayhem... [a] terrible sight."
He said people were calling out for their friends and calling for help. A member of the crew and a passenger from the ferry he was on dived into the water to help, he said.
"The boat was in pieces. It was a chaotic scene," Marshall said.
Television pictures showed helicopters shining searchlights on to the waters of the harbour. Police divers and a navy vessel were also searching for the missing girl.
It was the second fatal accident in two months involving ferries in one of the busiest parts of the harbour. In January a ferry ran into a small fishing boat, also near the bridge, critically injuring a fisherman, who later died.