Nicholas Springate, BBC's Jerusalem bureau chief, said the BBC crew who had been filming a Palestinian doctor as he made house calls, were detained and made to sit for four hours at gunpoint by Israeli troops on Thursday.
The soldiers demanded the journalists hand over phones, confiscated their videotape and denied their requests to be allowed to contact their bureau or Israeli authorities.
"The BBC will be issuing a formal statement and complaint," Springate said.
Israeli military officials said the BBC journalists and two Palestinians had unwittingly walked into a building where an Israeli undercover operation was under way to hunt for resistance fighters.
Officials said the undercover troops had been afraid Palestinian fighters in the streets might get wind of their presence in the building, and had put the journalists under guard and taken away their phones until the operation ended to maintain secrecy.
"Unfortunately there was no other way the soldiers could have acted without exposing themselves and endangering themselves," said Israeli army spokeswoman Major Sharon Feingold.
The Israeli army said it had apologised to the BBC for the incident and had launched an investigation.