Blair also said that Britain needed a wider debate on Muslim integration, and particularly over the need to strike a balance between integration and multiculturalism.
"It is a mark of separation and that's why it makes other people from outside the community feel uncomfortable," he said at his monthly press conference on Tuesday, pointing out that UK Muslims should be free to dress as they wished.
"No-one wants to say that people don't have the right to do it. That's to take it too far."
Blair was speaking in reference to a recent case when a British school fired a Muslim teacher for refusing to take off her veil for classes.
"We need to conduct this debate in a sensitive way, but it needs to be conducted"
British Prime Minister
The school said that she was unable to teach while showing only her eyes.
Blair has previously been reticent over the issue, but on Tuesday said he supported the handling of the case.
"I can see the reasons why they came to the decision that they did," he said.
Blair also said that the controversy over the veil highlighted the difficulties faced by British Muslims trying to balance their loyalty to their state and to their religion.
"People want to know the Muslim community in particular, but actually all the minority communities, have got the balance right between integration and multiculturalism," he said.
"We need to conduct this debate in a sensitive way, but it needs to be conducted."
Jack Straw's comments on veils angered many Muslim women
He later added that the veil debate was part of a wider struggle within the Muslim world over the future of Islam.
"There's a second issues which is about Islam itself, and how Islam comes to terms with, and is comfortable with, the modern world," he said.
His comments came after Jack Straw, a cabinet minister and former foreign secretary, wrote two weeks ago that he asks Muslim women who come to his constituency surgery to remove their veils.
Straw later added that he would prefer if women did not wear veils at all.
The recent controversy has focused mainly on the right of Muslim women to cover their faces - rather than on their right to just cover their hair.