Television footage showed a special forces police officer shooting the already-wounded man several times at close range as he lay in the street in front of the building.
Reports said the man was trying again to detonate the bomb while on the ground.
Police cordoned off the area in front of the Justice Ministry and the suspect could be seen on the ground as a member of the bomb squad, wearing protective clothing, removed the man's shirt and then held up a tan coloured cylinder with wires coming out of it.
The demolitions expert defused the bomb about 10 minutes later.
The shooting took place in one of the busiest parts of Ankara, the ministries neighbourhood, in a crowded park abutting the city's main road, Ataturk Boulevard.
The TV film showed the bomber on the ground with his head moving. Reports said he died soon afterwards.
Police identified the dead man as Eyup Beyaz and said he was a member of the Revolutionary People's Liberation Party-Front, or DHKP-C, an outlawed leftist group that aims to topple the government and replace it with a Marxist one.
The group has been responsible for several bombings in the past.
CNN-Turk television said Beyaz was wanted by police for planning two attacks: on the wedding of Prime Minister Recep Tayyip Erdogan's son in August 2003, and during a Nato summit meeting in Istanbul last year.
The bomber allegedly belonged to
an outlawed Turkish leftist group
Beyaz first tried to enter the Justice Ministry through the staff entrance and was stopped when metal detectors went off, reports said.
Security personnel tried to search Beyaz, who refused to submit to the inspection, the Anatolia news agency said.
Police then tried to handcuff him, at which point Beyaz hit the detonator on his homemade bomb, Anatolia said.
The bomb failed to explode, but the detonator made an exploding sound that startled police, Anatolia said. Beyaz, taking advantage of the confusion, ran out of the building.
Police first shot him in the leg and he collapsed on the ground. Anatolia said that while on the ground he tried again to set off the explosives.
A police officer could then be seen shooting him repeatedly.
Initial reports identified the bomber as Muharrem Akyurt, the name that police said was on the false ID card he was carrying.
When asked by reporters whether he may have been the target of the attack, Justice Minister Cemil Cicek refused to comment.
"No, no, no, I won't say anything on the issue," he said.