After breaking even with Guatemala in the sixth round, the tenth round produced only 77 votes for Venezuela compared to 110 for Guatemala. Five nations abstained.
Neither country has reached the two-thirds majority needed for victory in the 192-nation UN General Assembly.
The vote has become a battle for influence between the US and Venezuela, which under US foe Hugo Chavez has tried to form an alliance in Asia, Africa and the Middle East to challenge Washington's interests.
Failure to get onto the UN Security Council would represent a set-back for Chavez's ambitions for a bigger international profile.
Chavez ruffled many diplomatic feathers when, in a September UN speech, he called George Bush the "devil" and said the American leader had left the smell of sulfur hanging in the chamber.
Votes of conscience
Further rounds of balloting are necessary during which a new compromise candidate could emerge. The highest number of ballots occurred in 1979 with 155 rounds in a contest between Colombia and Cuba, with Mexico then emerging as the compromise candidate.
Francisco Javier Arias Cardenas, Venezuela's UN ambassador, said the US had tried to turn the vote into a contest between his government and Washington, and said votes cast for his government had been "votes of conscience" in favour of the developing world.
"We are not competing with a brother country. We are competing with the biggest power on the planet," he told reporters, adding that Venezuela would not withdraw from the race.
The Security Council has 15 seats, five permanent members with veto power - the United States, Russia, China, Britain and France - and 10 nations serving for two-year terms, five of them elected each year.
In other regions, South Africa, Indonesia, Italy and Belgium received the necessary votes to win two-year seats in the council, beginning on January 1. They replace Tanzania, Japan, Denmark and Greece. It was first time ever South Africa will have a seat on the council.
Venezuela and Guatemala are vying for the Latin American seat being vacated by Argentina while Peru stays on the Security Council until the end of 2007 along with the Congo Republic, Ghana, Qatar and Slovakia.
Christian Wenaweser, Lichtenstein's UN ambassador, told reporters: "It's going to be very hard to turn that [vote] around."
He and several other ambassadors said that Chavez's September "devil" speech to the UN General Assembly did not win him friends.
"Many people felt it was bad taste," said Augustine Mahiga, the Tanzanian ambassador. But he said Guatemala might have won the seat outright had the US not lobbied so hard on its behalf.
Within Latin America, nations have been divided, with Argentina, Brazil, Paraguay, Uruguay and Bolivia considered supporters of Venezuela. Mexico, Colombia and most Central American nations have indicated support for Guatemala.
Most decisions in the Security Council, the most powerful UN body are made by the five permanent members. But policy statements need the support of all 15 members and a resolution has to have a minimum of nine votes and no veto for adoption.
Roy Chaderton, the Venezuelan envoy who played a key role in lobbying on behalf of Chavez's government, said the results were only a minor setback in a prolonged struggle against US efforts to dominate international affairs.
Chaderton told state television after US-backed Guatemala topped Venezuela in the first four rounds of voting: "Swimming against the current isn't easy. It can be tiring at first, but the more one swims, muscles get stronger.
"This battle will prepare us for another battle within the international community."
Francisco Arias Cardenas, Venezuela's ambassador to the UN, said earlier that Washington was using "brutal pressure" through threats and blackmail to keep Venezuela off the council.
He said: "Many countries still don't dare to follow the steps Venezuela is making.
"We are going to prepare for the next round, and fight with dignity."
Chaderton accused the US government of "using all its power" to try to thwart Venezuela's chances.
He said: "There were a lot of telephone calls made from Washington yesterday to capitals on various continents to coerce and scare countries that had decided to vote for Venezuela."