The Muslim community in Britain is asking why the men were arrested in the first place, and, then, released without being questioned about the alleged plot.
After the men were freed in the early hours of Wednesday, Gareth Peirce, their lawyer and a veteran human rights campaigner, said they had been given no explanation.
"Not a word was ever mentioned to either of them about a plot to kidnap or the grisly suggestion of a beheading or even of a soldier at all," she said.
Mohammed Shahim Akhtar, general-secretary of the UK Islamic Mission, told AFP: "If we have more incidents like this, it's going to antagonise youths in particular."
Senior Muslims say the raids in Birmingham and central London, with the current atmosphere of distrust, is fuelling radicals seeking to stir up young people.
Previous raids also prompted protests in east London after a man was shot and wounded in a police operation in Forest Gate.
A district judge gave police extra time on Tuesday to question seven of the men detained, but told police that they would have to either charge or release the other two.
Shahim Akhtar said that he feared the raids were "arming" radicals and "playing into their hands," and urged police to develop better community relations.
"I'm fearful they may be a reaction which may not be in the way that we like," he said.
"I'm fearful they may be a reaction which may not be in the way that we like"
Shahim Akhtar, general-secretary of the UK Islamic Mission
Jafer Qureshi, from the Muslim Council of Britain umbrella group of organisations, told BBC Radio that community relations in Birmingham had been left "in tatters".
Moazzam Begg, a friend of one of the men and who was detained for two years at Guantanamo Bay, told AFP: "There's a huge amount of scepticism within the community.
"I believe that they will probably all be released and then the credibility of the police, of the media, amongst not just the Muslim community will be severely tested."
British human rights group Liberty has criticised what it called unattributed briefings given to reporters in London on what the men were said to have been planning.
It said it suspected political motives in the leaks to divert attention from criticisms of John Reed, the home secretary, in a row over prison overcrowding.
Meanwhile, West Midlands police issued a statement defending the release of the men as "normal and to be expected in large, complex criminal inquiries where a number of arrests have taken place".
Ruth Kelly, Britain's communities secretary, also launched on Wednesday a $9.9m fund for local politicians to use to counter the "false and pernicious" ideology of Muslim extremism.