Livni declared victory in the early hours of Wednesday, saying: "Today the people chose Kadima... We will form the next government led by Kadima."
She appealed to rival Netanyahu to join a national unity government led by her.
But Netanyahu said he should become Israel's next prime minister because right-wing parties had won enough votes for a governing majority.
"With God's help I will lead the next government," Netanyahu told his party members shortly before Livni spoke to hers.
"I will turn to our natural partners in the nationalist camp [to form a coalition government]."
Overall, hawkish parties appeared to have won a majority of seats up for grabs, giving Netanyahu the upper hand in forming the next government.
Wednesday's tally did not include thousands of votes cast by soldiers. They could shift the results slightly when they are counted by Thursday evening.
Yuval Steinitz, a Likud Knesset member, told Al Jazeera that Netanyahu would be asked to form a coalition because "Kadima and Livni can't form a government. Two or three months ago she had more seats together with the Labor party and she failed".
Under the electoral system, the president will ask the person deemed most likely to get a working coalition to form a government, rather than the candidate with the most seats.
The chosen party will be given 28 days - a period which Shimon Peres, the president, may extend by up to 14 days - to form a coalition holding at least 61 of the 120 seats in the Knesset or parliament.
Lieberman said his party's strong showing meant that he now held the key to forming the new Israeli government.
United Arab List
He said he had spoken to both Livni and Netanyahu and told them he could be persuaded to join either one of them.
Menachem Hofnung, a professor of political science at Hebrew University, said "it is up to Lieberman who will form the next coalition".
"Lieberman has emerged as the kingmaker. He is the winner of this election, and it depends on who he sides with over the next few weeks as to who will be prime minister."
Israel's press declared that Netanyahu was the more likely candidate to be able to rally the other parties behind him.
"He lost but he will conquer," the Maariv newspaper said.
The Haaretz daily said: "Despite the poll results, it is not certain that Livni will be able to muster the 61-seat coalition needed to form a government."
Shlomo Ben-Ami, a former Israeli foreign minister, told Al Jazeera that neither Livni nor Netanyahu could claim a "majority which could rally the Knesset", and criticised the election system as "an obstacle to solving the problems for the nation and the conflict with Palestinians and the Arab world".