The tower, set to open in 2011, will be the tallest of the five skyscrapers planned to replace the trade centre, which was destroyed by the al-Qaeda attack on September 11, 2001.
New York ceremony
As construction workers, politicians and architects applauded, a massive crane lifted the first, column, which was painted with an American flag and the words "Freedom Tower" and set it over steel bars on the southern edge of the tower's base.
By next spring, 27 of the giant steel columns will anchor the skyscraper and rise to street level from the bottom of ground zero.
Michael Bloomberg, the mayor of New York, said: "Rising from the heart of the World Trade Centre site, the Freedom Tower will symbolise the spirit of our city and our nation: inspiring, soaring and undefeated."
Lengthy negotiations over who would build the tower and security concerns have delayed the project.
Politicians laid a granite cornerstone in July 2004 to begin construction, but had to move the building after city police said it was too close to traffic, making it vulnerable to terrorism.
Eliot Spitzer, the governor-elect who takes office next month, has said he planned to look again at designs for the tower.
Federal and state agencies, including the governor's office, have agreed to occupy half of the building's office space.
The tower will be built with 45,000 tonnes of steel, builders say.