World food prices are at an all time high due to fuel price hikes, the switching of the production of food products to bio-fuels and commodity speculation.
Addressing the food crisis, Annan said that in the short term "we need to provide enough resources for organisations like the WFP [World Food Programme] and others to provide assistance to those in need.
"In the medium-term we need to take steps to improve food supply on the world market," and stop governments from sitting on stocks.
"For the long-term, and particularly in Africa, we need a green revolution that will help in farmers to increase production. Africa is the only continent that can not feed itself."
Annan said that African governments and the private sector must assist in major investment in agriculture.
Zanele Twala, the South African country director of Action Aid who was also at the meeting, said that G8 countries must take a large share of the blame for the decline of African agriculture.
"Aid to African small scale farmers has dwindled to half its 1980 levels while direct payments to farmers in G8 countries topped US$125bn in 2006," Twala said.
"The advent of the bio-fuel industry has seen more and more agricultural land being set aside for bio-ethanol production."
The group, talking at the Nelson Mandela Institute, said that food provision needed to be seen as a human right that should be part of governance.
Mary Robinson, the first female president of Ireland, said that if legislation were to guarantee the provision of food then systems would be to make governments accountable for the delivery of food to the poor.
"It is not acceptable that the poor have another crisis on their hands that we can change," she said.
Archbishop Tutu said that "the right to food is fundamental".
"The world today is captivated by an obsession with fighting a war against terror. We are wasting time and resources.
"If there is one war we can predict we will never win it is the war against terror, as long as we make the conditions that make people desperate."
The elders panel was created in 1999 as a group of world leaders convened to help resolve national and international issues.
The group also includes Jimmy Carter, the former US president, and Aung San Suu Kyi, the Myanmar opposition leader.
This year is the 60th anniversary of the universal declaration of human rights, signed by UN member states as a common standard of achievement for all peoples and all nations.