Karim Tuanaya, acting head of the local port authority, said on Thursday the ship's engine broke down after encountering five-metre-high waves.
He said rescuers were losing hope of finding more survivors, especially with rain and huge waves hampering the search.
"The problem is that the sea remains rough with waves up to four metres high, while the weather has kept visibility very low," he added.
Tuanaya said the 29 people who had life vests on were plucked from the sea by a passing oil tanker, which also pulled out the bodies of two children from the sea and is assisting in the rescue operations.
Indonesia has suffered a series of air and sea accidents in recent months, and low safety standards have been blamed for most of them.
In late December last year, a passenger ferry carrying around 600 people sank in a storm in the Java Sea, killing more than 300 people.
In February, another ferry caught fire northwest of the capital, Jakarta, and later sank with investigators and journalists on board.
The government has said it is considering new regulations banning old ships and aircraft from service.