Security forces said they raided 12 houses in Srinagar on Wednesday morning and shot dead Nasir Mahmud Ahwan, a Pakistani who was deputy commander in India of Jaish-e-Muhammad.
A police statement said Ahwan served as the "right-hand man" of Gazi Baba, who was killed in a raid in Srinagar on August 30.
Since Baba's death, violence has soared in Indian-administered Kashmir, with at least 205 people killed.
Police were tipped off to Ahwan's presence late on Tuesday when two men led them to "12 well-constructed and concealed hideouts of Jaish-e-Mohammad".
Troops from the Border Security Force and police headed to one of the hideouts when they were ambushed, setting off a two-hour gunbattle in which Ahwan was killed.
According to police, Ahwan, who was from the eastern Pakistani province of Punjab, "was involved in about 20 killings of security force or police personnel".
Jaish commander Baba was accused of masterminding a series of attacks against India, including the December 2001 raid on India's parliament building in Delhi which set off war tensions between India and Pakistan.
Kashmir is claimed by Pakistan,
India and Kashmiri separatists
Jaish-e-Muhammad was founded in Pakistan, but banned there a month after the Indian parliament attack.
In a separate incident, police sources said two soldiers from the Indian army's counter-insurgency wing were killed on Wednesday morning in an ambush by fighters near Kangan township, 40km northeast of Srinagar.
More than 38,000 people have died in Indian-administered Kashmir since the eruption of fighting in 1989, according to Indian officials.
However, the fighters put the toll between 80,000 and 100,000 dead.