Five heads of state and officials from a number of other AU members signed the convention, which still needs ratification by the individual countries.
Signatories included Robert Mugabe, Zimbabwe's president, and Sheikh Sharif Ahmed, the president of Somalia - which has Africa's third largest IDP population after Sudan and the Democratic Republic of Congo.
Museveni called on AU states to ratify the pact, which needs 15 of the AU's 53 members to do so before it comes into force.
Julia Joiner, the AU commissioner for political affairs, likewise urged leaders to "turn intention into action".
"This instrument clearly demonstrates that African leaders are conscious of the difficulties that displaced persons experience and are poised to, as much as possible, put an end to their suffering," she said.
The convention integrates the main aspects of international humanitarian law and would oblige AU states for the first time to prevent displacement, come up with solutions to the causes of displacement and provide IDPs with basic rights.
The Norwegian Refugee Council (NRC) and other human rights and aid agencies welcomed the pact.
"The African Union is the first regional organisation worldwide to adopt a legally binding instrument to protect the rights of internally displaced people," Elisabeth Rasmusson, the NRC secretary-general, said.
"The convention adopted by the heads of state in Kampala today is the opportunity for the continent to be at the forefront of ensuring that the rights of displaced populations are upheld."
Delegates said the convention underwent last-minute changes to an article on the obligations to IDPs of armed groups in African countries, after several countries - including Zimbabwe - raised concerns that the original wording gave legitimacy to such groups.