Fishing is a key sector of the country's economy and Icelanders are concerned that the EU's common fisheries policy would give other European fleets access to the nation's rich waters.
"I realise that we have two difficult issues to deal with, the fisheries and agriculture," Skarphedinsson said.
"I think it could be difficult negotiations ... But I think within three years we can become a member."
Skarphedinsson, a former fisherman, said Iceland would not accept a "rotten deal" for its fishing industry.
"To be frank with you, if we would get a rotten deal on the fisheries, the Icelandic people would get quite angry."
Iceland would have to hold a referendum on membership before joining the EU. A poll in May showed that just over 60 per cent of the Icelanders supported the apllication.
Iceland is already part of the European Economic Area, a trading block that gives Icelanders the right to live and work in the EU, while the country is allowed to run its own agricultural, fishing and monetary policies.
Iceland, with its population of about 320,000, would become the EU's smallest member if its entry is approved.