His brother, Zhanybek, accompanied him, according to Russia's ITAR-TASS state news agency.
Other Russian news agencies reported that he would have talks with Nursultan Nazarbayev, the Kazakh president.
Prevent civil war
The interim government called his departure a "deportation" in a statement. It said that Bakiyev had submitted a request to resign and reports added that Baktybek Kaliyev, a former defence minister and a key Bakiyev ally, had been arrested. These claims were not confirmed by Bakiyev's side.
The interim government also said it would later seek Bakiyev's transfer to a Kyrgyz or international court for trial.
Since being ousted, Bakiyev had been holed up in Jalal'abad where he maintains support.
Kazakhstan, which chairs the Organisation for Security and Co-operation in Europe, confirmed Bakiyev had left the country and said it was an important step towards preventing civil war.
It said that joint efforts between themselves and Dmitry Medvedev and Barack Obama, the Russian and US presidents respectively, had allowed for the agreement for Bakiyev to leave the country.
Kazakhstan also said that the move would help to bring stability and the rule of law to Kyrgyzstan.
Al Jazeera's Robin Forestier-Walker, in southern Kyrgyzstan, said: "It is a happy outcome for those who were looking for a diplomatic solution to the problem."
Earlier in the day bodyguards of Bakiyev fired shots into the air in an attempt to disperse opposition protesters who gathered to disrupt a rally he was holding in the southern city of Osh.
The shots were fired shortly after Bakiyev took to the podium to address around 2,000 supporters.
"This was a result of Bakiyev venturing out further afield to Kyrgyzstan's second city of Osh," Forestier-Walker said.
"When he went there he was hoping to be greeted by hundreds of thousands of supporters, but instead he was heckled and felt threatened ... he was forced to beat a hasty retreat.
"It was an indication that he was beginning to realise that his continued insistence on holding on to the presidency was creating more destabalisation in the country."
The interim government has called for Bakiyev to surrender, saying he should face trial for "spilling blood" during the unrest last Wednesday that left at least 80 people dead.
Bakiyev fled to Jala'abad during the anti-government protests and has said he will only step down if the interim government guarantees his and his family's security.
No arrest warrant has been issued for the president, but one is in effect for Zhanybek, the former head of the state guard service.
Bakiyev has denied ordering troops to fire on the protesters in the capital.
Forestier-Walker said that the Russian and US governments have pledged funding to the interim government within the last 48 hours or so.
"It is certainly being treated as a government, it is being given that legitimacy. Because it is trying to put the country back on its feet," he said.