'Head in hands'
Brown later visited Duffy, a grandmother, in her own home in Rochdale to apologise.
"Sometimes you do make mistakes and you use wrong words, and once you've used that word and you've made a mistake, you should withdraw it and say profound apologies, and that's what I've done," Brown said, after spending more than 30 minutes with the pensioner.
Earlier, the prime minister was shown on television with his head in his hands as the comments were replayed to him during an interview with the BBC.
The incident, a rare unscripted moment in a highly choreographed campaign, may undermine his attempts to reclaim lost ground on Thursday in the last of three televised leaders' debates which will focus on the economy, seen as Brown's strongest card.
Andrew Russell, a politics lecturer at Manchester University, said: "A politician in a stronger position could recover from this.
"What we know is that Gordon Brown is not in that position ... I don't think it's a good idea to call voters bigots."
Bookmakers widened their odds against Labour winning a majority in the May 6 election to 46-1 after Brown's gaffe, from 30-1 before.
Duffy challenged Brown on a number of issues, including east European immigration
"There's no doubt in what the punters are saying: Brown has royally messed up this time," said Mike Robb, a spokesman for online betting company Betfair.
Opinion polls on Wednesday continued to point to a hung parliament, in which no one party wins an overall majority.
But they showed the Conservatives, the main opposition party, and Labour had regained some ground from the Liberal Democrats, the traditional third party in British politics.
The Lib Dems have enjoyed a strong ratings boost since the first TV leaders' debate, helping to turn the contest into a three-horse race.