Al Jazeera's Gabriel Elizondo, reporting from Manaus, said that the meeting had been thrown together at the last minute.
"They apparently got some confirmation over the last few days that the heads of state would be coming, but then at the last minute several of them pulled out saying that their schedules would not allow it," he said.
"This is definitely a very embarrassing issue for Lula."
Relations between the leaders of Venezuela and Colombia are poor due to a dispute over Bogota's agreement to allow US forces greater access to military bases in the country.
A series of border incidents, including the destruction of two footbridges by Venezuela, have led Colombia to last week to put its forces on "maximum alert".
"There has been a lot of sabre-rattling been going on between those two countries," Elizondo said.
"So I think once Hugo Chavez said he wasn't going to come, then Uribe decided he was going to go either. I think there was a bit of political one-upmanship."
Environmental activists gathered in Manaus before the talks to urge leadership at the climate summit in Copenhagen.
"We are sending a message to Obama, Lula and Sarkozy. Actually, it is for all world leaders. There is no more time for talks," Paulo Adario, Greenpeace's Amazon co-ordinator, said.
"Either they take on a historic role to fight and make something concrete in Copenhagen and go down in history or they will go to waste."
Brasilia has pledged to voluntarily cut its greenhouse gas emissions by at least 36 per cent by 2020, while the US announced roughly 17 per cent reductions for the same year.
Deforestation in the Brazilian Amazon has fallen to its lowest level in 20 years, but still 7,000sq km of rainforest were still cleared.