Footage on local television showed wrecked taxis and destroyed shop fronts as residents carried wounded people to hospitals minutes after the blast.
General Freddy Padilla, the head of Colombia's armed forces, blamed fighters from the Revolutionary Armed Forces of Colombia, known as Farc, a Marxist armed group that has been battling the government in Bogota for more than four decades.
"Surely this was the Farc," he said.
But Alvaro Uribe, Colombia's president, said it was too early to lay blame as police had not yet announced the type of explosive used.
Uribe announced a reward of up to $150,000 for information leading to those responsible.
"We can't let our guard down," he told reporters in Medellin before heading to Buenaventura.
Guillermo Mendoza, Colombia's chief prosecutor, said he would not rule out retaliation against his office by drug traffickers, or a fight between rival gangs.
No group has claimed responsibility for the blast.
Buenaventura is Colombia's largest port, handling half of Colombia's coffee exports, but one of the country's poorest cities and a major drug trafficking route to the Pacific coast.
Uribe, who leaves office in August, has put the Farc on the defensive since taking office in 2002 and it is estimated their forces have been halved, to about 6,000-10,000 fighters.