ASWJ's spiritual leader, Sheikh Mahmoud Sheikh Hassan, said the groups would need financial support from the international community to integrate their administrative, leadership and military structures and fight al-Shabab and Hizbul Islam.
"This is not a fight or struggle against people but against an ideology," he said at the ceremony.
"The meaning of this agreement is to save the people of Somalia and the reputation of the Islamic faith."
Ancient Sufi tradition
Somalia has a Sufi tradition going back more than five centuries. The country's Sufis have been angered by the desecration of graves, the beheading of clerics, and bans on celebrating the birth of Prophet Muhammed imposed by the hardline Wahhabi fighters.
Al-Shabab, which professes loyalty to al-Qaeda, is battling to impose its own version of sharia (Islamic law) throughout Somalia.
The government Sharif Ahmed, Somalia's president, has agreed to implement a more moderate version of sharia.
Omar Abdirashid Ali Sharmarke, the Somali prime minister, said that under Monday's deal, Ahlu Sunna would be given five, as yet undetermined, posts in the cabinet and would appoint deputy commanders of the military, the police and the intelligence services.
"This agreement is a victory for peace and a crushing defeat for extremist groups," he said.
"This day will go into history as the day of peace for the Somali people and the region as a whole."
The government has said for several months it will launch a major offensive but has yet to do so. Rebels have stepped up attacks in various parts of the capital this month.
Hizbul Islam said Ahlu Sunna would just lose support by joining a government which has little influence outside Mogadishu and is often criticised as being corrupt and divided.
"The Addis agreement will not have any positive impact. It will only lead to the destruction of the so-called Ahlu Sunna," Sheikh Mohamed Osman, Hizbul Islam spokesman, told the Reuters news agency.
However, a resident in the Ahlu Sunna-held town of Dusamareb in central Somalia said the deal brought some hope.
"The power sharing deal is likely to reduce Somalia's chaos," Ali Suleiman said by telephone.
"We now smell peace: if the government and Ahlu Sunna have united, the rebels will be pushed from opposite sides and thus weakened."
In Dubai and also on Monday, Somali Islamist politicians, thinkers and scholars held a conference attended by Sharif Ahmed.
The conference called on warring factions in Somalia to lay down arms immediately and start direct talks.
The attendees urged the anti-government fighters to join the government and parliament and leave the "nation's [religious] scholars" to mediate and solve their country's crisis.
A recent report released by the UN blamed Eritrea for supporting anti-government fighters in Somalia, but also described Somalia’s government forces as "corrupt, undisciplined and inefficient".
At least 21,000 Somalis have been killed since the start of 2007, 1.5 million have been uprooted from their homes and nearly 500,000 are sheltering in other countries in the region.