In the speech on Thursday, Abbas said: "We have to bring down this bunch, which took over Gaza with armed force, and is abusing the sufferings and pains of our people."
An Abbas aide said the president was not calling for a popular revolt against the group, rather reacting passionately to the bloodshed in Gaza City.
Despite Abbas's words, he has little influence in Gaza where Hamas has taken new steps to cement its grip on the territory since the rally.
It has rounded up more than 400 Fatah activists and announced media restrictions and plans to limit public gatherings.
Hamas condemned Abbas's comments and accused him of selling out the Palestinian people in peace talks with Israel.
A formal relaunching of negotiations is set to take place later this month at a US-sponsored summit that is also meant to bolster Abbas in his struggle with Hamas.
In a televised speech on Thursday, Ismail Haniya, the deposed Hamas prime minister, said: "Hamas is hard to break, and it is beyond being wiped out or eliminated. We are people whom without us things can't be finalised."
Haniya said there would be an "honest, fair and transparent" committee to investigate the events at the rally, and that Fatah activists arrested there would be released, except for those directly involved in inciting violence.
Five months of Hamas rule and crushing international sanctions have taken a heavy toll on already impoverished Gaza, and the hardship helped drive the Fatah rally, which drew about 250,000 people on Monday.
Abbas's call for overthrowing Hamas coincides with his efforts to bridge differences with Israel ahead of the summit in Annapolis, Maryland.
Abbas said in his speech that his government was "working relentlessly" to make the summit a "decisive-launching pad" for establishing a Palestinian state.
But he demanded that Israel halt all settlement construction, release Palestinian prisoners and end its killing of suspected Palestinian fighters.
Abbas was speaking on the 19th anniversary of the Palestinian declaration of independence at a meeting in Algeria.