Pakistani police have begun investigations into the incident in which 15 police officers were killed.
Television footage showed wounded security personnel being taken away and ambulances rushing to the area.
Dead and injured police officers could be seen lying on the ground in pools of blood, their blue uniforms ripped to shreds by the force of the explosion.
"We were playing cricket in a nearby park when we heard a deafening blast. There were several policemen on the ground and me and my friends took them to hospital but they were dead," Shaqeel Ahmed, a witness, told the AFP news agency.
Al Jazeera's Kamal Hyder, reporting from Islamabad, said that the explosion happened on the edge of the security cordon.
People were leaving the mosque after the mostly peaceful rally when the attack targeting the police occurred, he said.
|No one has claimed responsibility yet [EPA]
Thousands of police had been deployed across the capital for Sunday's rally and a tight ring of security was in place around the gathering in front of the mosque.
Barbed wire and fences had been put up to prevent vehicles from entering the area while pedestrians had to pass through metal detectors.
Our correspondent said the bombing was likely to jeopardise the government's attempts to engage the leaders of armed groups carrying out attacks in the country.
There was some evidence to suggest that there were splinter groups that were not in favour of rapprochement with the government, he said.
Yousuf Raza Gilani, Pakistan's prime minister, condemned the attack and ordered an inquiry, according to state media.
"Such incidents are against the teachings of Islam and do not serve any purpose," he was quoted as saying.
Imtiaz Gul, a political analyst in Islamabad, told Al Jazeera that the attack came after threats from the Pakistani Taliban in northwest regions bordering Afghanistan.
"Beitullah Mehsud had indicated and threatened that if his people, with whom the government was talking ... if they don't get and fair and if their prisoners are not released ... they would take revenge," he said.
"They clearly indicated it would be in Islamabad, Lahore or another big Pakistani city."
There were no immediate claims of responsibility for the blast.
Pakistani troops surrounded the mosque last July, clashing with pro-Taliban and alleged al-Qaeda members who were holed up inside before storming the building.
More than 100 people were killed in the military operation.
'Rooting out terrorism'
Pervez Musharraf, the president who ordered last year's assault, reiterated the government's "commitment to root-out terrorism in all its forms and manifestations", the state-run Associated Press of Pakistan news agency said.
He urged Gilani's government to do more to combat anti-government groups, warning that otherwise there would be "Red Mosques everywhere".
The level of violence in Pakistan has fallen since last year but attacks are still relatively frequent.
In June, a suicide car bomber killed at least six people near the Danish embassy in Islamabad.
A statement attributed to al-Qaeda claimed responsibility for that blast, which was believed to have targeted Denmark over the publication of cartoons depicting the Prophet Muhammad.