"We want to have them back safe and sound. Their children, spouses, parents, siblings and loved ones are anxious and waiting for them to come home."
The ICRC said it has not heard from the three captives since Monday.
Jesus Verzosa, head of the national police, said he has received information that the victims are unharmed but he declined to give more details.
"We are negotiating and we're trying our best to ensure the safety of the hostages as we have always been doing in other incidents," he said.
Police are co-ordinating closely with soldiers who are searching jungle territory in the interior of southern Jolo island for signs of the ICRC workers and their captors, he said.
Julasirim Kasi, police chief on Jolo, told Reuters news agency that the kidnapping was believed to be led by Albader Parad, a senior leader of the Abu Sayyaf group.
Abu Sayyaf, which has been linked to Jemaah Islamiyah (JI) and is believed to be sheltering people suspected of involvement in the 2002 Bali bombings, has a notorious reputation for kidnapping.
Jolo, a majority Muslim island about 950km south of Manila, was once portrayed as a success story in the Philippine military's battle to flush out the Abu Sayyaf but the group has since resumed kidnappings and decapitations.
Meanwhile, the military's relations with the locals have deteriorated due to operations that have forced villagers from their homes.
Despite American military training and assistance, including dozens of US troops stationed in Jolo, Abu Sayyaf fighters have abducted several people on Jolo and nearby Basilan island in recent months.
So far, all the abductees have been released, usually after a ransom has been paid.
Philippine officials have said they believe the group is resorting to ransom kidnappings because it is running out of funds.