Dulmatin, a bomb-making specialist, was killed in a raid on an internet cafe in a suburb of the Indonesian capital on Tuesday.
Two other alleged JI fighters were killed in a separate co-ordinated raid nearby, Indonesian police said.
The men were suspected of supporting a new group, calling itself al-Qaeda Indonesia, who clashed with police last week in a raid on their training camp in the northern province of Aceh.
|Dulmatin was killed in a police raid on a Jakarta internet cafe on Tuesday [EPA]
"Preliminary investigations show that these people are related to the terror suspects that are still being chased in Aceh," Major-General Edward Aritonang, an Indonesian police spokesman, said in Jakarta.
"These people were providing funds and weapons."
Dulmatin is believed to have been one of the key planners behind the 2002 Bali bombings which killed 202 people, including 88 Australians.
He is also thought to have been behind several other high-profile attacks carried out by JI.
Greg Barton, a regional security expert who teaches at Monash University in Melbourne, told Al Jazeera that Dulmatin's death could unravel the JI.
"The capacity of that network has suffered a serious loss. The fact that he has been on the run for eight years and is now out of action, I think, is very good news," he said.
According to Indonesian police, Dulmatin was trained in Afghanistan by al-Qaeda and until recently was thought to have fled to the Philippines.
In 2008 Philippine security forces found a body which they believed at the time to be Dulmatin, although his identity was never confirmed.
The US government's Rewards for Justice programme has offered a bounty of up to $10 million for his capture.
It was not immediately clear whether anyone will be eligible to claim the reward.
Speaking before Yodhoyono's confirmation of Dulmatin's death, Australia's prime minister had praised Indonesia's counterterrorism forces for what he said was their difficult and dangerous work.
"The breakthroughs which Indonesia has made in undermining various terrorist networks have been significant," Kevin Rudd said at a joint news conference with the Indonesian leader.
The death of the man alleged to be JI's top bomb-maker comes just two weeks before the US president, Barack Obama, is due to make his first visit to Indonesia.
Indonesia's struggle against terrorism and US support for its counterterrorism forces is expected to be one of the key issues for discussion during the visit.