At least three people - a Hamas gunman and two police officers - were injured during the fighting early on Friday, witnesses said.
Tensions arose when the 3,000 strong Hamas-backed force, formed under the authority of Said Siyam, the interior minister, deployed across the territory on Wednesday.
A day later about 1,000 Palestinian police officers marched through Gaza in a show of allegiance to Mahmoud Abbas, the Palestinian president and the leader of the Fatah faction.
After a tense stand-off, the Hamas force surrounded the main police station and parliament building in Gaza City and exchanged fire with those inside.
Frightened residents rushed from the streets as gunfire echoed around the city early on Friday.
A policeman who had brought an injured colleague to hospital, said: "It began with the two sides shouting at each other and it developed into a gunfight."
Siyam had announced a plan last month to create the new force, which is controlled by the interior ministry and headed by Jamal Abu Samhadana, a resistance fighter wanted by Israel.
Abbas vetoed the plan and ordered Hamas to remove its forces from the streets.
Hamas has refused to disband
its forces saying they are legal
Hamas has refused to do so, saying the unit is legal and suggesting it would not be disbanded.
Ismail Haniya, the Palestinian prime minister, vowed on Friday not to disband the new security force and said he was prepared to increase its size in defiance of Abbas.
"We do not intend to make one step backward. The force will stay and will be intergrated into the police force. Their task is to protect internal security and if there will be a need to increase its number, we will do it," Haniya told worshippers at a mosque in Gaza City.
Tensions between Hamas and Fatah have risen since the Islamists swept to power in January's elections.
Meanwhile, Condoleezza Rice, the US secretary of state, said Palestinian civilians were suffering from the dangerous situation created by the rival forces.
"It's a very tense situation and one that we hope will be resolved," Rice said at a Washington press conference.
"We obviously will leave that to President Abbas, who we believe has the confidence of the Palestinian people and should be able to exercise his responsibilities as president of the country," Rice said.