The Brotherhood, an Islamist movement banned in Egypt but which says it is committed to political reform by peaceful means, won a fifth of the seats in parliament last November and December, putting the ruling party on the defensive.
The members stood as independents because the government does not recognise the Brotherhood and has refused to let it form a political party, on the grounds that it would be based on religion.
Ahmed Nazif told Reuters in an interview: "Islamists who say they belong to an illegal organisation have been able to go into parliament and act in a format that would make them seem like a political party ... We need to think clearly about how to prevent this from happening."
He said the government could not take away the right of individual citizens from running for parliament, but members of the Brotherhood were different. "We have a secret organisation represented in parliament. They are not individuals," he said.
The prime minister's remarks were another indication that the Egyptian government is having second thoughts about concessions it made to the political opposition last year when it was under pressure from the US.
In recent weeks Egyptian police have taken a much tougher approach to street protests. Plainclothes security men have beaten, kicked and clubbed people demonstrating peacefully in support of judges demanding independence from the executive.
Nazif dismissed accounts of attacks on protesters and said that, since there were other channels for expressing dissent, only "thugs" would take to the streets.
"Why blame the police? I am frankly fed up by the fact that people are blaming those who are trying to keep the peace against the people who are trying to break the peace," he said.
Earlier on Saturday Nazif said his government was not in a hurry to change the country's political system.
No way back
"It doesn't take a month or two or six. It will take years ... We have the time. We are not in a hurry," he said.
He said that after the electoral success of the Islamists, the government had to think again about the course of reform.
"You need to recalculate, you need to revisit some of your assumptions, to make sure you are really on the right track but in the end I don't think there is any way to go back on this."