Khaled Marghlani, the health ministry spokesman, said the four victims were all suffering from health problems already, including cancer and respiratory illness.
Three of the deaths announced on Saturday were in Medina and the fourth was in Mecca itself, the health ministry statement said.
Marghlani said 16 other cases of swine flu infection among pilgrims had been detected, and that "four are in hospital in critical condition".
At least 20 people suffering from the disease had recovered completely after treatment, he said.
Reporting from Mecca, Al Jazeera's Ayman Mohyeldin, says the deaths have raised alarm bells.
"Officials are working to ascertain whether these individuals arrived in the kingdom already infected with swine flu or if they got the virus when they arrived.
"There are expected to be at least 3 million pilgrims attending it [the Hajj] coming from all countries around the world, but the concern for the officials is how to prevent the outbreak", Mohyeldin said.
Face-masks have been distributed to the general public and disinfectants are readily available. Authorities are also asking people not to eat in public areas to preserve hygiene.
Saudi health authorities have mobilised for the world's largest gathering since swine flu began spreading across the globe.
In Medina and Mecca, as well as in Jeddah, the government has taken measures to identify the virus as early as possible.
Thermal cameras have been installed at the air and sea terminals where most pilgrims arrive. At least 15,000 health workers are deployed, and hospitals have hundreds of extra beds available.
The health ministry also has mobile units which can instantly send to a central monitoring centre the locations of infections, to monitor outbreaks.
The World Health Organisation released data revealing that at least 6,750 people had died from swine flu worldwide since the virus was first uncovered in Mexico and the United States in April.
The peak day in the hajj will be on Thursday, when pilgrims gather at Mount Arafat, where the Prophet Muhammad gave his final sermon.