Defence Minister Shaul Mofaz spoke a day after a senior Hamas activist returned to Gaza after 15 years of exile, using the Palestinian-operated crossing.
It was not clear whether Mofaz's threat was linked to the return of Fadel Zahar, brother of Hamas leader Mahmoud Zahar.
Fadel Zahar had been deported by Israel to Lebanon in 1991. On Tuesday, he entered Gaza through the Rafah terminal on the Gaza-Egypt border, which is operated by Palestinians under the supervision of European monitors.
Israel has expressed concern that Palestinian resistance fighters and weapons would enter Gaza once the Palestinians are in charge of the border.
Israel withdrew from the coastal strip in the summer and closed Rafah as part of the pullout. Rafah reopened last week, as part of a US-brokered deal on new border arrangements.
Hamas spokesman Sami Abu Zuhri said he did not know whether other Hamas leaders planned to return from exile.
"They have the right to return to their home whenever they want"
Sami Abu Zuhri,
"They have the right to return to their home whenever they want," Abu Zuhri said.
Speeding up the flow
As part of the arrangements, Israel is also to expedite the flow of Palestinian passengers and cargo at the Karni and Erez crossings between Gaza and Israel.
"If it does not improve and the Palestinians do not cooperate (in Rafah) we will close the Erez and Karni crossings," Mofaz said on Wednesday, during a tour of the area.
"They will become international crossings in all senses and I really hope that the Palestinians understand the significance of this step," Mofaz said.
Such a step would mean the West Bank and Gaza would no longer be part of a customs union with Israel, a step that could devastate the struggling Palestinian economy.