The incident in the western state of Gujarat came a little over a week after suspected Muslim militants set off two bombs in the holy Hindu city of Varanasi, killing 15 people and wounding many more.
Police said that they were conducting routine searches in Gujarat's main city of Ahmedabad after midnight when they were shot at from a house and returned fire.
The state police chief said of the dead men: "Two of them are Pakistanis and the other two Kashmiris."
Another police officer said the men were from the Harkat-ul-Mujahideen militant group fighting Indian rule in disputed Kashmir.
The identities of the men were established through some leaflets, diaries and mobile phones found in the house, police said. Pistols and explosives were also found.
"They had planned to bomb religious places and also train other terrorist cells," a police spokesman said.