The two men were reportedly detained in Malaysia for having falsified travel documents.
Indonesian authorities have also said that the two alleged leaders of the armed Indonesia-based group are suspected of being involved in attacks in the eastern island of Sulawesi.
The island's Christians have been targeted several of those attacks, including the beheadings of three schoolgirls in 2006 and a bombing at a market that killed seven people.
A police official told Reuters the arrests could help lead to the capture of Noordin Mohammad Top, a Malaysian who is one of the most senior members of JI.
The Southeast Asian group, which allegedly linked to al-Qaeda, has been blamed for a string of bombings and attacks in Indonesia and elsewhere in the region.
Sidney Jonesform the the International Crisis Group said that unconfirmed reports indicated Abdul Rohim may have been the acting JI leader after its former head was arrested last year.
She said Agus, a doctor from East Java, is believed to be heading the group's operations in Sulawesi.