Those problems were highlighted this week by the difficulty in getting consensus among Asean's 10 member states on the wording and scope of the pact.
For example, some Asean members expressed reservations over the inclusion of the word "families" in the draft, fearing a conflict with domestic laws and other social concerns.
Indonesia, the Philippines, Laos, Cambodia, Vietnam and Myanmar are major exporters of cheap labour to Malaysia, Singapore and Brunei.
Labour groups such as the Philippines-based Migrant Forum in Asia say Asean has yet to show it has the political impetus and legal framework to turn the pact into reality.
"Asean has to move away from the principle of non-interference if it wants to resolve the challenges"
Sinapan Samydorai, Taskforce on Asean migrant workers
William Gois, its regional co-ordinator, said the proposal must go beyond rhetoric to firmly address a host of issues that plague millions of migrant workers.
"Governments will need to be consistent in prosecuting those responsible and ensure the victims have access to redress mechanisms and a safe environment which encourages greater reporting of abuses," he told Al Jazeera.
"The problem for Asean is to come up with a document that will be taken seriously and incorporate labour standards including freedom of association, the right to organize and bargain collectively, and eliminate all forms of discrimination at the workplace."
Gois said Asean should recognise that its people were not "tradeable commodities".
Another group based in Singapore said the proposed pact was a political gimmick.
Sinapan Samydorai, convenor for the Taskforce on Asean migrant workers, said the Philippines had come up with the idea to gain political mileage on home ground.
He believed the pact would have no impact as the countries hosting migrant workers were unlikely to implement most or any of the recommendations.
"Asean has to move away from the principle of non-interference if it wants to resolve the challenges," he said.
Emphasising the lack of a regional framework, Edmund Bon, a Malaysian human rights lawyer, said the focus should be on the type of issues and not the type of victims.
The latter blurs the lines of protection and can be counter-productive, he said.
What is needed, he said, was a legal framework promoting a progressive culture of human rights protection for all.
Without that, Bon said, adopting resolutions or making declarations would do little to change the lives of migrant workers.