Two office buildings and one workers' dormitory collapsed as a result of the blast, reports said.
Rescuers lowered listening devices into the rubble to search for survivors, but the reports said there were no signs of life under the debris.
The official Xinhua news agency said the factory, which produced quartz sand, was owned by Jingxin Mining, a private company.
The explosion at the factory was caused by the illegal stockpiling and storage of five to seven tonnes of explosives on the premises, Xinhua said.
"The enterprise is just a quartz sand processing factory, not a mining business, and has no need for explosives," Wang Tuqiang, the city's vice-mayor, was quoted as saying.
The factory owners were detained by police, the Xinhua report said.
China's work safety record is uneven and thousands of people die every year in mines, factories and on construction sites, according to official sources.
Official figures show that more than 3,200 workers died in mines last year in China, with accidents regularly attributed to the illegal storage of explosives.
Independent observers, however, say the actual figure could be much higher, as many accidents are covered up.