Delgado, who declined to give his name because he was not an official police spokesman, said officials had recovered 21 bodies and rescued 18 people who were hurt in the collapse on Saturday morning.
Jairo Amaya, the civil defence director, said: "Talking to local residents in this region, they said there were about 60 people working here when the mine collapsed. So far we have accounted for 54."
Rescue efforts have been hampered by the lack of any registry or count of how many people entered the mine system.
Ana Salcedo, wife of missing miner, said: "As far as I know, my husband is trapped in the mudslide, because so far nobody has seen him, nobody has taken him out. I went to the hospital to look for him and he is not on the injured list either. So people say he must still be trapped in the mine."
Rich in resources, but with a limited government presence across much of the country, makeshift mines are a common sight in Colombia, in particular those zones where gold or emeralds have been found.
With little or no oversight, mining accidents are a frequent occurrence.
In February, 32 miners died at a coal mine in northeast Colombia after an explosion.