The shock result prompted the government's chief peace negotiator to offer his resignation as the leader of the Revolutionary Armed Forces of Colombia (FARC) vowed on Monday that the rebels were prepared to "fix" the deal.
UN Secretary General Ban Ki-moon, who had offered a United Nations team to oversee the disarmament process, said he "urgently" sent his Colombia envoy to Havana, Cuba, where the peace talks were held, for new consultations.
Still, the outcome left no clear way forward to end a half-century conflict that has claimed more than 260,000 lives.
"Colombia is in uncertain territory," said Angelika Retteberg, an expert on the peace process at the University of the Andes, in Bogota.
The peace deal had been hailed as historic from the time it was concluded on August 24 to the moment it was signed last week in the presence of Ban and US Secretary of State John Kerry.
Resentful of the blood shed by the Marxist rebels and the lenient punishment the deal meted out for their crimes, voters rejected it on Sunday by a razor-thin margin: 50.21 percent for the "No" camp to 49.78 percent for "Yes."
"Any mistakes we made are my responsibility alone," Humberto de la Calle, the head of the government's delegation to the talks, told journalists at the presidential palace.
"I have come to tell the president that I am offering to stand down as head of the delegation, because I will not be an obstacle to what comes next."
That could pave the way for new negotiations with a government team including hardliners from the party of Santos' top political rival, former president Alvaro Uribe, who led the "No" campaign.
FARC leader Rodrigo "Timochenko" Londono said the peace deal remained in effect despite Sunday's popular vote rejecting it.
"The [vote] does not have any legal effect. The effect is political," he said on Colombian television.
Santos admitted defeat, but pledged not to give up. "I will continue to seek peace to the last day of my term," he said.
Santos had been tipped as a top contender for this year's Nobel Peace Prize, along with rebel leader Londono. But that prospect is all but dead after the referendum defeat, experts said.
The deal has now "lost legitimacy", said Maria Luisa Puig, a Latin America analyst at the Eurasia Group consultancy.
"The deal ... is dead and cannot be implemented," she said. "While the parties may seek to renegotiate a deal, the process would be complex, time consuming, and probably outlive the current administration."
The FARC launched its war in 1964, after the army crushed a peasant uprising. Over the years, the conflict drew in several leftist rebel groups, right-wing paramilitaries, and drug gangs.
Authorities estimate it has left 45,000 missing and nearly seven million uprooted.