Straw, the leader of the lower house of the British parliament, said on Friday that the face veil was "a visible statement of separation and difference" and that he felt more comfortable dealing with women who uncovered their faces in his constituency.
Writing in the Lancashire Evening Telegraph, Straw said he had received a unanimously positive response to his request for Muslim women to take off their face veils - known in Arabic as the niqab - when they came to see him in Blackburn, the northern English town he represents in parliament.
The Islamic Human Rights Commission said that Straw was "selectively discriminating".
Other Muslim groups said they were less concerned by the comments.
Daud Abdullah, of the Muslim Council of Britain, said he understood Straw's views.
He said: "There are those who believe it is obligatory for the Muslim woman to cover her face. Others say she is not obliged to cover up. It's up to the woman to make the choice."
Straw, a minister in the ruling Labour party, said he defended the rights of Muslim women to wear headscarves and that wearing a face veil "breaks no laws".
He acknowledged that his concerns could be "misplaced".
"I thought a lot before raising this matter a year ago, and still more before writing this. But if not me, who?" Straw said.
David Cameron, the opposition leader, said many communities were growing up living "parallel lives" and that only contact could overcome differences.
The Muslim Association of Britain (MAB) accused both political parties of pandering to anti-Muslim elements.
"Both the Labour and the Conservative politicians are trying to compete with each other. One wonders whether they are trying to appeal to the right-wing media," a MAB spokesman said.
Abdul Hamid Qureshi, chairman of the Lancashire Council of Mosques, said Straw would be criticised "from all quarters".
"What is [Straw] he really concerned about? This is not helpful, it has got the potential to cause anger," he told the Lancashire Telegraph.