"When there's one, that's all right. It's when there are a lot of them that there are problems."
Hortefeux has denied that he made any racist remark, and on Thursday moved to account for his remarks.
"We cracked a few jokes about his origins, my attachment to Auvergne, and about the fact that I could speak Auvergnat [a dialect spoken in France] but I mentioned that I could not ... that a few pictures would be fine but I couldn't stay for more because I needed to leave," he said.
"That's all it was. And once again, not a word from me makes any allusion to a geographic community or origin," Hortefeux told reporters on Thursday.
Segolene Royal, a former Socialist Party presidential candidate, said that Hortefaux "has to apply to himself what he is preaching to others", in reference to his sacking of Paul Girot de Langlade, a senior French prefect who was accused of making racist remarks.
Patrick Lozes, president of the CRAN (representative council of black associations), said Hortefeux had not done "anything to make us think [his remarks were] taken out of context".
But Benjamin Lancar, the president of the UMP youth movement, said that he was "backing Brice Hortefeux and the young man involved".
"Above all, we do not accept that such a controversy took place. For me, the fight against racism is an essential fight. But it is tarnished by polemics such as this one," he said.
The young man, who has been identified by French media as Amine Benalia-Brouch, made his own statement in which he said no racist remark was made.
"This video has created a buzz and a polemic around Brice Hortefeux ... But he didn't make any insulting remark on Arabs or anyone else," he said.
Racial discrimination against minorities sparked riots led by black and north African young men around France in 2005.