Somali government security official Mohamed Nur said their forces overran the rebels' base in the same neighbourhood as the al-Shabab fighters attacked them on Wednesday.
At least 23 civilians were killed in Wednesday's clashes.
Nur said: "We have taken control of the position of the terrorists where they used to arrange their attacks. Our forces, getting assistance from the African peacekeepers, are now gaining military momentum in northern Mogadishu.
“The enemy completely emptied their positions here in northern Mogadishu after being forced to retreat. Three of our soldiers are injured so far."
Government forces have been planning a wide offensive to dislodge the insurgents from Mogadishu, where the rebels have confined the Western-backed government of President Sharif Sheikh Ahmed to just a handful of zones.
Residents reported seeing AU tanks moving into their neighbourhood.
A witness named Said Yusuf said: "Two military tanks and six other armoured vehicles belonging to the AU forces were at the frontline ... and one of their bulldozers was filling the trenches dug by the insurgents in the street."
The AU-backed government forces and the al-Shabab fighters have been locked in a tense stand-off since the government announced plans in January to dislodge the radical Islamists from Mogadishu.
In January, Mogadishu residents started fleeing ahead of the planned government offensive but the assault never came and some civilians have started returning to the capital.
Since taking control of much of Mogadishu after bloody clashes last year, the rebels have repeatedly carried out many attacks against foreign peacekeepers and the government troops, inflicting heavy casualties.
Civilians have borne the worst brunt of the relentless fighting, many of them caught in crossfire or killed by mortar shells fired in retaliation to attacks by the opposition who operate in residential areas.
Al-Shebab, who control 80 per cent of south and central Somalia, vowed to topple the internationally-backed government, which owes its survival to the African Union forces.
Somalia has been wracked by two decades of bloody violence sparked by the ouster of President Mohamed Siad Barre.