The building has since been surrounded by government troops where thousands of Zelaya supporters have held demonstrations since his return.
At least one person has been reported killed and several wounded in a series of clashes between Zelaya's supporters and security forces.
More than 100 people have been detained.
'Violence not necessary'
Speaking by telephone to Al Jazeera, Zelaya said he believed he had enough support in the Honduran military to return to power, but denied he would use violence to do so.
|Riot police have surrounded the Brazilian embassy following a series of clashes [AFP]
"Violence is not necessary," he said. "Violence is a tactic of the weak. When you use violence you have no other means."
Shortly after talking to Al Jazeera, Zelaya reportedly held a meeting in the Brazilian embassy with the four leading presidential candidates contesting presidential elections scheduled for November.
They are urging the ousted president and the interim government to reach a deal to resolve the political crisis.
"What we have asked ... is that they be very flexible in a dialogue," National Party candidate Porfirio Lobo told reporters after a meeting with interim President Roberto Micheletti.
Zelaya has been demanding talks with Micheletti since he returned to the country earlier this week.
Micheletti's de facto government has not yet released any comment on the talks, although earlier on Thursday officials announced the lifting of a nationwide curfew imposed following Zelaya's return to Tegucigalpa.
On Tuesday, Micheletti's government said it was willing to talk to Zelaya if he recognised the legality of November's presidential elections.
Willing to talk
Micheletti had repeatedly threatened to arrest Zelaya if he tried to return to Honduras and had insisted that Brazil hand over the ousted leader to "pay for the crimes he committed" which he said included corruption and violating the constitution.
But on Tuesday Michelletti said he was willing to talk to Zelaya if the ousted president recognised the legality of presidential elections scheduled for November.
He also said he would "talk with anybody anywhere at any time, including with former President Manuel Zelaya".
On Friday the UN Security Council is expected to discuss the crisis which has gripped Honduras since Zelaya was overthrown in a June coup.
The meeting comes after Spain's prime minister, Jose Luis Rodriguez Zapatero, earlier called for Honduras' interim leaders to restore democracy to the country.
"We won't accept the coup," Zapatero told world leaders gathered at the United Nations General Assembly meeting in New York.