Karzai's campaign also said the president did not want to take part because all the country's 41 presidential candidates were not invited.
Ghani, also a former senior World Bank official, has repeatedly asked Karzai to debate with him.
"It is the Afghan public that will suffer another broken promise ... if the future plans of each candidate is not made clear standing side-by-side his or her rival," Ghani said on his website earlier this week.
Jahid Mohseni, the chief executive of Moby Group, which owns the Persian-language Tolo TV, said the station hoped to hold a series of three debates and that Karzai was welcome to join the remaining two.
Mohseni said negotiations over Thursday's debate began more than three weeks beforehand.
He said: "Afghanistan's obviously a new democracy and we've got a lot of limitations in terms of communication and road structure.
"We saw a TV debate as useful because it crosses barriers in terms of literacy and candidates can talk to voters in their own homes."
'Show of weakness'
Before the debate, Haroun Mir, the deputy director of the Afghan Centre for Research and Policy Studies, said Karzai knew very well that if he took part in the debate, he would "face criticism from the two other candidates".
"For the past five years, as leader of Afghanistan, he has not been successful. It will be very difficult for him to defend what he has accomplished in the past five years. That's why he is trying to avoid direct debate," Mir told Al Jazeera.
Ajmal Habidy, Ghani's aide, said: "If President Karzai is not participating in the debates, it shows his weakness."
Though Karzai's popularity has slipped in recent years, no challenger is expected to be able to defeat him in the election unless they combine their campaigns and back a single candidate.