Guatemala, whose candidacy had been backed by the United States, received 109 votes in the first round, then 114, 116 and finally 110.
Venezuela's chances appeared to fade as the voting proceeded, but then jumped at the end, having received 76 votes, 74, 70 and then 75.
The result opened the door for others to join the race, in what could be a blow to both countries' chances for a seat on the council.
Now diplomats will search for a compromise candidate to break the deadlock, though they were also waiting for either Guatemala or Venezuela to drop out of the race.
But, both nations said they would remain in the race.
Venezuela's UN envoy said on Monday his country won't give up trying for the seat for 2007-2008 despite losing to the US-backed Guatemala in the first rounds of voting.
Francisco Arias Cardenas complained that the United States has pressured countries worldwide to prevent Venezuela from winning the rotating seat, which president Hugo Chavez hoped to use as a platform to speak out against US "imperialism".
"We are going to continue and we are going to call on countries of dignity, strength, independence and autonomy, which is what the United Nations needs right now"
Francisco Cardenas, Venezuela's ambassador to the UN
"We are going to continue and we are going to call on countries of dignity, strength, independence and autonomy, which is what the United Nations needs right now," Arias Cardenas said.
Guatemala has the support of Colombia, apparently most of Central America, Europe and other countries.
Some diplomats had expressed concern that Washington's support might actually hurt its bid by turning the contest into a US versus Chavez battle.
Venezuela has served four times on the Security Council.
Guatemala, emerging from years of US-backed dictatorship, has never had a seat but is a leading contributor of troops to UN peacekeeping missions.
The 10 non-permanent seats on the council are filled by the regional groups for two-year stretches.
The other five are occupied by the veto-wielding permanent members: Britain, China, France, Russia and the United States.
The 192-nation General Assembly elected South Africa, Indonesia, Italy and Belgium to the four other open seats in the council.
They will start their terms on the council on January 1, replacing Tanzania, Japan, Denmark and Greece.