"On my view of this matter, a mistake has been made," he said.
"To me the primary decision to make was to determine whether or not this prosecution was on sound footing. I've made that determination.
"I'll now take further steps to inquire how that mistake was made."
Haneef is alleged to have given a mobile phone SIM card to a relative who was later allegedly involved in last month's failed plot to bomb central London and Glasgow airport.
He denied any involvement with the failed attacks.
The Australian Federal Police (AFP) have been criticised for their handling of the case.
"It is a complex and painstaking process and the AFP will continue to work with its UK colleagues to fully explore the evidence and establish the facts"Mick Keelty, Australian federal police commissioner
Commenting after the charge was dropped, Mick Keelty, the Australian federal police commissioner, said the decision did not change the grounds for cancelling Haneef's visa.
"This remains an ongoing investigation," he said. "It is a complex and painstaking process and the AFP will continue to work with its UK colleagues to fully explore the evidence and establish the facts."
The immigration minister earlier this month had cancelled Haneef's visa, following his arrest on July 2, based on information given by the AFP.
It is not immediately clear whether Haneef will face deportation or be allowed to remain and appeal against the visa cancellation.
Haneef's lawyers said they would oppose any move to deport him, the Australian Broadcasting Corporation reported.