The military said soldiers spotted the three crawling towards the border fence near the Kissufim crossing point late on Sunday carrying a bag, allegedly planning to plant a bomb, and they opened fire.
Palestinian officials said the three were killed, and contacts were under way with the Israelis to turn over their bodies.
It was the most serious incident of its kind since Israel completed its Gaza pullout last month.
The clash comes as Palestinian negotiator Saeb Erikat was to sit with Dov Weisglass, a top aide to Israeli Prime Minister Ariel Sharon, a day before the scheduled summit between Sharon and Palestinian President Mahmoud Abbas.
It was not immediately known whether the talks would proceed.
Although both sides have reasons to hold a well-publicised summit to symbolise continued progress towards peace after Israel's pullout from Gaza and part of the West Bank, Israel has balked at making gestures it said would harm its security.
Chief Palestinian negotiator
Palestinians said there was no point in a summit just for the pictures - concrete results were needed.
Abbas himself came under criticism in 2004 when he was prime minister under the late Yasser Arafat, for meeting Sharon but coming away empty-handed.
Abbas is to travel to Washington later this month to meet US President George Bush, and he would be reluctant to arrive without any concrete achievements from a meeting with Sharon.
The Palestinians continue to demand the release of prisoners Israel is holding, the handover of West Bank towns to Palestinian control, the removal of army roadblocks and the reopening the Gaza-Egypt border crossing.
Israel appeared prepared to discuss the border, remove some roadblocks and free some prisoners, but not enough to satisfy the Palestinians.
An Erikat-Weisglass meeting on Sunday ended without agreement.
Dov Weisglass (R) is a top aide
to Israeli PM Ariel Sharon
Sharon told his cabinet on Sunday that he did not know if the meeting would take place, stressing that he would not make moves that he believed threatened Israel's security.
"There are gestures that Israel can and will make, and there are gestures that it won't," he said, according to the meeting participants.
Abbas has said one of his goals will be to persuade Israel to carry out pledges it already has made, including pulling its troops out of five West Bank towns.
Israel agreed to the pullback as part of a February ceasefire agreement. But the process stalled after two towns, Jericho and Tulkarim, were handed over, with Israel accusing the Palestinians of failing to take action against fighters in those towns.
Israel later retook Tulkarim after a bomber from the area attacked an Israeli city.
Defence Minister Shaul Mofaz opposes any further handovers of Palestinian towns, the Defence Ministry said on Sunday. Mofaz also objects to supplying Abbas' security forces with new weapons.
The Palestinians say their security forces need better equipment to restore law and order in the chaotic Palestinian areas.
Palestinians want Israel to
release prisoners it is holding
Abbas has resisted international calls to disarm resistance groups, fearing it would provoke civil war, although he recently banned armed groups from displaying weapons.
The fighters have repeatedly ignored the ban, although leaders of Abbas' Fatah movement called a news conference in Gaza late on Sunday to reaffirm their commitment to it.
The two sides have made progress on other Palestinian demands in recent days, with Israel showing flexibility on the release of some of the more than 7000 Palestinian prisoners it holds and a possible compromise on reopening Gaza's border with Egypt.
Israel closed the Rafah terminal - the main exit out of Gaza - before the withdrawal. The Palestinians say reopening the border is essential for Gaza's ravaged economy.
Israel reopened a cargo crossing on its border with Gaza on Sunday, and Palestinians said this would alleviate a shortage of fruit and dairy products in the territory.
The Karni passage and others with the Palestinian area had been closed almost continuously since 24 September after dozens of rockets were fired from Gaza into Israel.
Separately, three Palestinians were injured in clashes between Palestinian security forces and al-Nasir Salah al-Din Brigades fighters in Gaza, Aljazeera's correspondent in Palestine reported.