Palestinian police stood by as about 50 armed men from Hamas and the Palestinian Resistance Committees rammed a dump truck into the cement wall, knocking down several large slabs on Thursday.
The incident came hours after the Palestinian Authority pledged to take steps to shut the Gaza frontier, breached after Israel handed over its former Jewish settlements in southern Gaza to the Palestinians on Monday.
Struggling to impose order after the Israeli withdrawal, the Palestinian Authority acknowledged some light arms had been smuggled into Gaza and said the border would be shut on Friday.
Israeli media reports said the army may move troops to buffer the long-quiet border with Egypt, alarmed that Palestinian fighters might find it easier to bring in weapons without an Israeli military presence on the border.
Israel "cannot accept a continued situation where the border will be entirely porous. What is happening in Gaza in recent days is complete anarchy", Israeli Foreign Minister Silvan Shalom told Army Radio.
"I have given them instructions to prevent violations whatever they are"
Egypt has promised to patrol the 12km border. Israel has said it wants to control Palestinian movement there but the Palestinians have rejected any Israeli role.
Palestinian President Mahmoud Abbas, visiting the southern border town of Rafah, said he gave security forces orders to prevent further border infiltrations.
"I have given them instructions to prevent violations whatever they are," he said.
Abbas acknowledged that "mistakes had happened because of joy" over Israel's departure after a 38-year occupation of Gaza where 1.4 million Palestinians live, but said the Palestinians now hoped to restore order to the territory.
Palestinians are celebrating the
Israeli pullout from Gaza
In addition to the border breach at Rafah, Palestinian security forces had stood virtually powerless while armed men and ordinary Gazans stormed evacuated settlement enclaves this week, smashing synagogues and looting materials. Police managed to impose some degree of control starting on Wednesday.
Earlier on Thursday, Egyptian police brought in fence posts and barbed wire to reinforce a breach of the barrier. Cairo waived border controls as Palestinians celebrated the withdrawal of Israeli troops from Gaza, but had told people to return home by Wednesday night.
Tawfiq Abu Khoussa, a spokesman for the Palestinian interior ministry, said the Palestinians had slowed the flow of people crossing between Gaza and Egypt. Palestinians had confiscated hundreds of kilograms of narcotics at the border, he said.
Khoussa said Egypt had seized some weapons before they could be smuggled, and that Palestinian police were searching for other arms believed to have been spirited into Gaza.
Hundreds of people were still trekking both ways through the Salaheddin Gate in the divided town of Rafah late in Thursday afternoon, and Egyptian police had abandoned even cursory attempts to check on the travellers.
Mohamed Zorab, a 17-year-old Palestinian, said he had got across by telling the police he was Egyptian while Palestinians along other parts of the border said they had entered on Thursday without crossing Egyptian police lines.
"The border should be left open because we are one people," said Obeid Salama, an Egyptian of Palestinian origin with family on both sides of the line.