Angela Merkel, the German chancellor, had urged Latin American countries to distance themselves from Chavez, prompting him to accuse her of being from "the same movement that supported Hitler".
The Venezuelan leader has also again criticised Alvaro Uribe, the Colombian president, over the release of hostages held by the Revolutionary Armed Forces of Colomba (Farc) rebel group.
Chavez said on Thursday it "will be very difficult" to secure the release of hostages being held by the Farc while Uribe is in power.
Uribe has accused Chavez of providing aid to the Farc, and on Thursday Interpol, the global police agency, said documents on laptop computers found in a raid on a rebel base in Ecuador allegedly showed ties between Chavez and the Farc were genuine.
Chavez dismissed Interpol's findings as "ridiculous".
Al Jazeera's Lucia Newman in Lima said there was the possibility that more rows could erupt during the summit.
"The day is long and we are expecting at some point the presidents of Venezuela and Colombia will come to face to face and there may be some verbal shots fired."
The summit is also expected to discuss free trade agreements between Latin American states and the EU.
Free trade supporters like Peru are losing patience with critics like Evo Morales, the Bolivian president, who accused Peru and Colombia this week of trying to exclude his nation from talks between the EU and Andean countries.
"We can advance at different speeds, but let's advance," Alan Garcia, the Peruvian president, said on Thursday, adding that his country should be allowed to move faster with the EU.
Morales, himself a former coca grower, fears free trade deals could hurt poor farmers in his impoverished country.
"We want trade, but fair trade," he said in Lima.
Morales and Chavez were also set to attend a parallel 'peoples summit' held in Lima at the same time as the talks.
The summit was due to close with a joint declaration on global warming, the food crisis and efforts to stem the drug trade.