Zabiullah Mujahid, a Taliban spokesman, said the attackers were dressed in Afghan army uniforms.
The Afghan president left the area in an armoured convoy after his speech.
Extra police deployed
Security was a major concern in the weeks leading up to the conference. Extra police have been deployed throughout the capital, and journalists reported long delays at checkpoints on Wednesday morning.
The Taliban warned delegates to stay away in an audio recording released last month, saying that "the punishment for participating in the jirga is death".
"I think some Afghans... will say, if they can't even secure the area around the gathering that they've talked about for months on end, with the immense security preparations they have in place, what chance do they have of trying to secure the rest of the country?" Al Jazeera's James Bays, reporting from Kabul, said.
Mirwais Yasini, the deputy speaker of the Afghan parliament, called Wednesday's attack "a very big security blunder".
"It was a big shock for us [at the jirga] and for the nation," he told Al Jazeera.
"We were sure that the security was very good here and there would be no security blunder."
Dialogue with Taliban
Delegates at the traditional assembly of elders hope to reach an agreement on how the government should hold dialogue with the Taliban.
Yasini said dialogue is the only solution to the conflict in Afghanistan.
"We have to have jirga because there is no military solution to the Afghan conflict," he said.
"The war [has] expanded in reach to almost 20 provinces of Afghanistan ... We have to have an end to that and that will be a dialogue."
"We have to have jirga because there is no military solution to the Afghan conflict."
deputy speaker of the Afghan parliament
Analysts say the delegates - which were selected by the government, and include tribal leaders, politicians, and members of civil society - are likely to reach a broad agreement on engaging the Taliban.
The plan calls for the government to offer jobs to low-level Taliban soldiers who agree to stop fighting.
In his opening address, Karzai criticised the Taliban for bringing suffering and oppression to Afghanistan.
"The Afghan nation is looking at you," he said, addressing the delegates.
"They await your decisions, your advice, so that you can show the Afghan nation the way to reach peace, to rescue Afghanistan from this suffering and pain."
Barack Obama, the US president, has called the conference "an important milestone that America supports". European diplomats have also hailed it as a "crucial step to demonstrate national consensus".
Staffan de Mistura, the head of the United Nations mission in Afghanistan, said he was cautiously optimistic that participants in the jirga would agree on a deal.
"I believe they're tired of fighting... the Afghans are tired of a conflict that they will never win, that nobody else will ever win," he said.
But critics of Karzai's government, and many outside analysts, are sceptical that the conference will produce a detailed blueprint for reconciliation with the Taliban.
Karzai's main rivals have been excluded from the conference and representatives from the Taliban and groups like Hezb-i-Islami were not invited.
Abdullah Abdullah, Karzai's main rival in last year's presidential election, declined to attend the conference, saying the hand-picked delegates do not represent Afghan public opinion.
Elders in several provinces, including Helmand and Khost, say the most influential tribal leaders were rejected in favour of those loyal to the government.
The Taliban is also dismissive of the event. In a statement sent to news organisations on Tuesday, the group said the conference does not represent the Afghan people, and is aimed at "securing the interest of foreigners".
Human rights groups say the list of delegates is too male-dominated: Only 20 per cent of the conference attendees will be women. The number of women was increased after Hillary Clinton, the US secretary of state, warned Karzai that women were being ignored.