Nuri al-Maliki, the Iraqi prime minister, and politicians from across Iraq's political spectrum watched as a white-clad honour guard carried al-Ubaidi's coffin and the other containing his sister's husband, a close aide, who was among five other people killed in Friday's attack.
Police said a young man shot the politician twice in the head with a pistol before opening fire on worshippers and throwing a grenade. He was then killed by mosque guards.
The assassination could undermine efforts for reconciliation in the country, which holds a parliamentary election next January.
The vote will be a test of whether Iraq's feuding factions can live in peace after sectarian bloodshed erupted after the 2003 US-led invasion, which overthrew Saddam Hussein, the former president.
Maliki's Shia-led government has said attacks will intensify ahead of the vote, and blamed recent violence on al-Qaeda and other Sunni groups still waging a rebellion despite a sharp fall in fighting in the past year.
Al-Qaeda, which views Shia Muslims as heretics, has accused members of al-Ubaidi's Accordance Front of being traitors for taking part in Iraq's political process.
US combat forces are due to leave Iraqi cities by the end of June, and withdraw completely by 2012.
Washington hopes political rapprochement can be reached before they depart.
Adel Abdul-Mahdi, Iraq's Shia vice-president, said at the funeral that all members of parliament were under threat.
"We must stand as one against the repulsive terrorist groups who target every faithful and sincere official."
Al-Maliki has ordered an inquiry into al-Ubaidi's death.