"The post of prime minister has not been my aim," Gandhi told incoming MPs of her Congress party on Tuesday.
"I would follow my inner voice. Today, it tells me that I must humbly decline this post," Gandhi said in parliament.
Supporters immediately interrupted her with shouts of disappointment, briefly stopping her from speaking.
"My aim has always been to protect the secular foundations of our nation," Gandhi continued.
"We have waged a successful battle but we have not won the war. That is a long and arduous struggle and I will continue it with full determination."
"I request you to accept my decision and I will not revert," she said. "There is no question. It is my inner voice, it is my conscience."
Gandhi's Congress party ousted India's Hindu nationalists last Thursday in possibly India's biggest election upset. Short of an outright parliamentary majority, Congress has been discussing coalition arrangements with leftist parties, notably India's main communist bloc.
The Bharatiya Janata Party (BJP), stunned by its election defeat, renewed a campaign against Gandhi after the election, saying her foreign birth disqualified her from holding the office of prime minister.
However, the Indian constitution does not ban a foreign-born individual from holding such office.
But by bowing out, Gandhi defanged her opponents of the only political weapon they could use against her.
"By choosing to humbly decline to take up the prime minister's post she has played a masterstroke," the Indian Express said.
As the leader of the biggest bloc in Parliament, Gandhi had been expected to be named prime minister on Tuesday in a meeting at the presidential palace.
But the 57-year-old widow of murdered former prime minister Rajiv Gandhi left the ornate, colonial-era building empty handed, with the question of who will lead the world's largest democracy left hanging.
The prospect of the Italian-born Gandhi becoming prime minister had divided many Indians but her decision not to take the post will end that debate.
Reports in the Indian media suggested Gandhi is instead forwarding the names of Manmohan Singh and Pranab Kumar Mukerjee, both of whom were Congress finance ministers, as possible premiers.