Gert Rosenthal, Guatemala's foreign minister, made the announcement in a radio interview on Thursday.
He said: "We will continue the battle and, in time, when we are absolutely convinced that we cannot continue, we will then meet with our regional group and search for another candidate."
Guatemala and Venezuela each hope to inherit the Latin American seat on the Security Council after Argentina's term expires.
Venezuelan officials, meanwhile, sent mixed signals about a possible consensus candidate.
Alberto Muller Rojas, a Venezuelan general and top adviser to Hugo Chavez, the Venezeluan president, said "neither of the two parties [Venezuela and Guatemala] will get the necessary votes", adding that "the election of an outsider" was likely.
But Rojas added that Venezuela would "not withdraw its candidacy as long as Guatemala does not withdraw its own".
Francisco Arias Cardenas, Venezuela's UN ambassador, ruled out a withdrawal by either country.
"That Venezuela withdraws because it has fewer votes? It's not possible because it would be admitting the veto power of the United States," he said.
"That Guatemala and Venezuela withdraw to search for a third candidate? That's not possible."
The UN General Assembly resumed voting on Thursday to pick between US-backed Guatemala and US critic Venezuela.
In new rounds of balloting on Thursday, Guatemala continued to lead but again fell short of the two-thirds majority required to win the non-permanent council seat.
On Monday, one round of voting ended in a tie but Guatemala has led in all the other rounds.