The Thursday arrests came after police obtained telephone transcripts of calls made to another suspect, now held in Spain.
Federal police said Spanish court documents appeared to link Shaikh Muhammad Umran from Melbourne and Bilal Khazal from Sydney with Imad al-Din Yarkas - who is being held by Spanish authorities.
Prime Minister John Howard told Radio Australia the government had “known about these contacts for some time … the police have sought the permission already of the Spanish authorities to interview the man in Spain."
Australian Federal Police said it considered both the nationals to be "of interest" as part of 65 ongoing counter-terrorism investigations.
The men had been under observation for some time but never arrested, the police spokesman added.
Khazal, a former baggage handler at Sydney airport, had his passport confiscated last year as he was deemed a security threat by the government after being named in a CIA report.
He has previously denied being an al-Qaida operative.
Shaikh Umran told ABC he had never heard of Yarkas and laughed at suggestions he was a member of al-Qaida.
Nevertheless, Yarkas may face accusations of being involved in the 11 September attacks in the United States, which are blamed on Usama Bin Ladin's organisation.
Australia has toughened up its counter-terrorism laws and stepped up investigations in the wake of the 11 September attacks and last October's bombings on the neighbouring Indonesian island of Bali.
But so far only one person, Jack Roche, has been charged with conspiring to bomb Israeli diplomatic missions in Australia.