Those who died are thought to have been from the family living in the destroyed house, Jerry Sanders, the mayor of San Diego, said at a news conference on Monday.
The plane crashed about 3km from the Marine Corps Air Station in nearby Miramar as it was preparing to land, Ian Gregor, a Federal Aviation Administration spokesman, told The Associated Press news agency.
Earlier there were reports of a second pilot in the plane.
It remains unclear what caused the crash, and military officials would not comment on the reason for the incident.
|The fighter jet ploughed into
a residential area [Reuters]
The pilot had just taken off from an aircraft carrier offshore in the Pacific ocean as part of a training exercise, Lieutenant Katheryn Putnam, a spokeswoman for the air station in Miramar, told the Reuters news agency.
Witnesses also told local radio the pilot may have been trying to get the plane to crash in a large canyon nearby in a bid to avoid the school grounds.
One San Diego resident told the MSNBC television channel that the crash shook his house several streets away in the neighbourhood and that officials were telling residents to move back.
"I knew definitely it was an explosion. ... It was pretty strong," he said.
Local television showed plumes of thick white smoke rising from the crash site in the suburb of the University City area, while the charred remains of a building were also clearly visible.