A US state department spokesman called the Swiss decision "regrettable".
"The United States thinks that the rape of a 13-year-old girl is a crime," said PJ Crowley.
In a statement after his release, Polanski expressed "massive thanks" to his supporters.
Polanski, 76, who won a best director Oscar for his moving portrait of life in the Warsaw Jewish ghetto of World War II in The Pianist, was still at his mountain chalet in the ski town of Gstaad, where he had been held under house arrest.
The electronic foot bracelet that the Swiss have used to control his movements had been switched off, the minister said.
"This is not about qualifying a crime. That is not our duty. This is not about deciding on guilt or innocence," she said.
The minister said while Washington could appeal against the decision internationally, she did not expect that to happen.
The announcement follows months of uncertainty over whether Polanski would have to return to the US after having been arrested in in Zurich in September 2009.
Polanski was detained on arrival in the city where he had been due receive a lifetime achievement award at the Zurich film festival.
"It's an enormous satisfaction and a great relief after the pain suffered by Roman Polanski and his family," Herve Temime, Polanski's lawyer, said.
His arrest prompted an outcry in the global film industry and in some political circles in France, where he has been a long-time resident, with directors from Woody Allen to Martin Scorsese and Jean-Luc Godard expressing support for Polanski.
After a short jail stint, Polanski, who holds dual French and Polish citizenship, was put under house arrest in December 2009 at Gstaad while Swiss officials awaited the outcome of US legal proceedings.
Escape from US
Polanski pleaded guilty to having sex with the girl but fled the US on the eve of his 1978 sentencing because he believed a judge might overrule his plea and put him in jail for 50 years.
The director has lived in Europe ever since, facing the prospect of arrest the moment he set foot back on US soil while continuing his film career outside Hollywood.
Born to Polish-Jewish parents in 1933, his life was marked by a narrow escape from the Krakow ghetto and by the murder of his pregnant wife, actress Sharon Tate, by followers of cult leader Charles Manson in 1969.
Polanski is also known for classics such as Chinatown, which earned 11 Oscar nominations, and Rosemary's Baby.
He completed his latest film The Ghost, based on a Robert Harris' best-seller, while under arrest in Switzerland.