Ghazi Nasib al-Din, also known as Ghulam Rasul Dar, commander of operations for Hizb al-Mujahidin, was killed on Friday in a gun battle in Pattan, 27km north of Srinagar, the capital of Indian-administered Kashmir, said army spokesman Colonel Dharam Adhikari.
Some of his associates were also killed in the clash, the spokesman said.
Hizb al-Mujahidin, which is fighting for the region's merger with Pakistan, is one of the biggest rebel groups. Nasib al-Din was the number two in the group headed by Sayyid Salah al-Din, who is based in Pakistan-administered Kashmir.
Nasib al-Din 's killing came hours after India's Border Security Force (BSF) said it had killed the Hizb al-Mujahidin's third-ranking leader, Abbas Malik alias Abbas Rahim, in a separate gun battle on the outskirts of Srinagar on Thursday.
Hizb al-Mujahidin named a new commander on Friday to replace Malik as its deputy operational chief.
"Commander Mahmud Ghaznavi has been appointed the new chief military adviser after the martyrdom of Abbas Rahi," Hizb al-Mujahidin said in a statement issued in Muzaffarabad, capital of Pakistani-administered Kashmir.
Besides weapons and a satellite phone, more than one million rupees ($22,000) in cash was found near the site of Thursday's gun battle, BSF spokesman Tirtha Acharya told Reuters.
Hizb al-Mujahidin detonated a car
bomb near Srinagar on Thursday
"The slain militant was against the recent peace moves... and was probably making a nefarious plan to destabilise the ongoing peace process at this crucial juncture," a BSF statement said, referring to last month's talks between Indian and Pakistani leaders.
India, which accuses Pakistan of backing the revolt in Jammu and Kashmir, its only Muslim-majority state, has agreed to begin talks with Islamabad in February to resolve all outstanding disputes, including their row over Kashmir.
But some local newspaper reports said Malik had been arrested by security forces and shot dead in a "fake gun battle". Hizb al-Mujahidin has so far not commented on Malik's death.
Separatist violence has continued in Kashmir despite peace moves between the South Asian rivals who have fought two of their three wars over Kashmir.
Despite the latest killings, Indian Deputy Prime Minister Lal Krishna Advani on Friday said there had been a perceptible fall in the level of cross-border infiltration and was optimistic on prospects for talks.
Advani is scheduled to have talks with Kashmiri separatists next week. Kashmir's largest separatist alliance, the All Parties Hurriyat Conference, has been invited by New Delhi for talks on 22 January.
"While cross-border terrorism and infiltration of terrorists into India has continued over the years, the level of infiltration and the number of violent incidents in Jammu and Kashmir have perceptibly come down in recent months," Advani told state police chiefs.
Advani, who also holds the home ministry portfolio, said the "record number" of tourists and pilgrims in Kashmir was indicative of increasing normality in the region.
Indian officials say the revolt has claimed more than 40,000 lives so far. Pakistan and separatists put the toll twice as high.