"He is above Abu Dujana and was captured on the same day," Nasution told a news conference in Jakarta.
After the announcement police played a videotape of the man they said was Zarkasih saying he had been the acting head of JI since 2004.
Officials said Zarkasih controlled the group's operations across the whole of Indonesia, as well as being in charge of training JI leaders, controlling weapons and ammunition and managing assignments.
"The JI network still exists, they are still building their power continuously. They recruit, train and collect weapons and ammunition," Nasution said.
The International Crisis Group estimates Jemaah Islamiah, whose goal is to set up an Islamic state in Southeast Asia, has about 900 members.
On Thursday, police said they were stepping up their hunt for another suspect, Noordin Mohd Top, a Malaysian who has pledged allegiance to al-Qaeda.
Noordin appeared on a video in late 2005 threatening attacks against the US and its allies in the "war on terror".
He is allegedly one of JI's chief recruiters and has successfully evaded Indonesian authorities for years.
In the 2005 video Noordin claimed to be head of "al-Qaeda's Jihad Organisation in the Malay Archipelago", a previously unheard of group.