The man, a local member of the al-Aqsa Martyrs Brigades, an armed wing of the mainstream Fatah movement, was whisked into an unmarked car in the town of Rafah on Tuesday, witnesses said.
An official told Reuters "he was arrested for interrogation about involvement in the kidnapping".
The hostages, human rights worker Kate Burton and her parents, were freed on Friday after being kidnapped in Gaza by a previously unknown group that called itself the Brigades of the Mujahideen-Jerusalem.
Armed Palestinian men stormed into a government office in the Gaza Strip to protest the suspect's arrest, witnesses said.
Members of the al-Aqsa Martyrs Brigades forced their way inside the Interior Ministry in the town of Rafah, demanding the release of their colleague, after burning tyres and shooting. The office was empty of personnel, witnesses said.
Armed men also exploded a bomb beneath the Gaza-Egypt border fence on Wednesday saying it was to press for the release of the arrested suspect.
The bomb failed to puncture the concrete barricade, and the
armed men quickly fled the scene.
Gaza has seen a rash of kidnappings and other violence since Israel withdrew from the coastal strip in September after 38 years of military rule. The violence has threatened to delay a Palestinian parliamentary election on 25 January.