Phillips was taken hostage by the pirates after the crew of the Maersk Alabama, which was carrying 5,000 tonnes of UN aid, fought back.
Relatives said Phillips had volunteered to join the pirates in their lifeboat in exchange for the safety of his crew.
At one point, he tried to escape by jumping overboard but "didn't get very far," a US official said.
John Reinhart, president and chief executive of Maersk Line Ltd, said the FBI was looking into the hijacking and the crew would not be able to leave the Maersk Alabama until the investigators had spoken to them.
"Because of the pirate attack, the FBI has informed us that this ship is a crime scene," he said.
"[The crew] won't consider it done until the captain is back"
head of Maersk Line Ltd
"When I spoke to the crew, they won't consider it done when they board a plane and come home," Reinhart said.
"They won't consider it done until the captain is back, nor will we."
Phillips' lifeboat, which has drifted to within 32km of the Somali coast, is being monitored by US naval vessels.
Major Stewart Upton, a Pentagon spokesman, said negotiations with the pirates were continuing.
Abdi Gara, a pirate commander, said on Friday that Phillips would be moved from the lifeboat to a larger ship, making it more difficult for the US sailors hoping to rescue him.
"I'm afraid this matter is likely to create disaster because it's taking too long and we are getting information that the Americans are planning rescue tricks like the French commandos did," he said.
Garad was referring to the storming on Friday by French marine commandos of the Tanit, a yacht, in which Florent Lemacon, a French yachtsman, and two pirates died.
Italian boat seized
Hijackings are an ongoing problem in the busy shipping lanes off largely lawless Somalia. At least a dozen ships have been seized in the Indian Ocean and more than 200 crew members are being held hostage.
|Phillips was being held hostage in a lifeboat as his ship arrived in Kenya on Saturday [Reuters]
On Saturday, the pirate claimed another victim when they seized an Italian-flagged tugboat in the Gulf of Aden.
Sergio Carvalho, a lieutenant on the Portuguese naval veseel NRB Corte-Real, said the tugboat, which had a 16-member crew, issued a distress call before communications went silent six minutes later.
Andrew Mwangura, an official from the East African Seafarers' Assistance Programme, a regional maritime group, said the crew on the tugboat, which he said was operated from the United Arab Emirates, were unharmed.
Mohamud Muse Hirsi, the president of Somalia's autonomous Puntland region, said that giving in to the pirates' demands was not an option.
"We do not advocate for any ransom to be paid to the pirates and we support the French government, which uses force, while taking on the pirates," he said.
Puntland is reputed to be the home of many of the pirates.
Tens of millions of dollars in ransoms is believed to have been paid for the release of vessels captured in recent months.
Ali Abdullahi, a security consultant and risk management analyst specialising on Somalia, said the only way to tackle piracy is to address the political and economic problems there.
"A lot of the time, we're not talking about the core issues which have made piracy a major event in Somalia," he told Al Jazeera.
"There has been a lot of illegal fishing by international agents, toxic dump wasting on parts of Somalia, a lack of bad governance as well, all of which are the real causes of piracy."