Lars Aage Eldoey, who managed to escape with his wife from the top floor, said: "We first thought it was an earthquake."
Eldoey, who realised what had happened when he got out onto a balcony, said: "There were enormous rocks, not just rocks, half the hill had slid down into the bottom floors."
Smoke billowed from the bottom floors of the building, with sagging balconies and a contorted glass entryway.
Rescuers said five people who lived in the apartments were unaccounted for and possibly buried by the debris inside.
Kjell Kvenseth, police rescue leader at the scene, said: "We fear that the missing people are dead."
Fifteen people were taken to a hospital, two of them with moderate injuries, Magne Tjoennoey, Aalesund police operations leader said.
Tjoennoey said rescuers had not been able to enter the building, which was finished in 2003, to search for the five missing people.
He said: "We have to wait for the propane tank to burn out. That will take time. It's big, with about four cubic meters of gas."
Tjoennoey said police had not been able to confirm whether the five people reported missing were in the building, or had not been home at the time of the collapse.
He said: "We have not been able to find out where they are. As time goes by, we increasingly fear that they are in the building."
A helicopter, rescuers and search dogs from around the region were called to the scene, including the Norwegian unit of the United Nations International Search and Rescue Advisory Group.
Tjoennoey said geologists at the scene saw the risk of further rock slides as small.