Rescuers are searching for people still be trapped in the building which had 36 apartments and about 75 people are believed to have lived there.
Authorities said poor construction was the probable cause of the collapse.
Red Cross officials said 50 people, many of them wounded, had been rescued so far and sent by ambulance to hospitals across the city. The bodies of two had also been found by rescue workers.
Lagos state governor Bola Tinubu visited the site and vowed to find and prosecute those responsible.
"The developer cannot run away. He has property in Lagos. The government will seize it and prosecute him," Tinubu told a gathering crowd.
Prince Oniru, an infrastructure adviser to the Lagos state government, said the building was weak and blamed those who built it. "The material used in the construction was very poor," he said.
The four-story building was reduced to a two-story pile of rubble.
Several trapped people could be heard shouting from inside what was left of the building, witnesses said.
The Red Cross said power went out in the Ebute-Meta district on Tuesday evening and volunteers worked though the night with candles and flashlights.
Police tried to keep back thousands of people who crowded around the building looking for missing relatives and picking through the rubble.
Authorities launched an investigation into the cause of the collapse.
Dozens of people have died in similar disasters in Nigeria, especially in overcrowded Lagos, during the last four years.
Building experts have blamed the collapses on the use of substandard construction materials, disregard for regulations and non-compliance with development policies.