Officers swarmed on the Second World Financial Centre late on Tuesday afternoon and led out men in business suits, whisking them away in vans and cars.
Some of the men covered their heads with overcoats while others bowed to hide their faces from television cameras and photographers.
"It's currency fraud... securities fraud," said one agent at the scene of the arrests. "It's been a long investigation. The arrests have been ongoing today."
Vice presidents arrested
A Madison Deane and Associates Inc employee, who asked not to be named, said the FBI arrested seven people at his firm, which offers currency broker services.
"We were just sitting there working, and they [FBI] just... took out three partners, three vice presidents and one broker all in handcuffs."
The worker said the FBI told Madison Deane employees that $4 million had been stolen from clients and that money had been taken out of people's Individual Retirement Accounts.
Madison Deane officials are not prepared to comment, but FBI sources said the Wall Street traders would be charged with money laundering and fraud for swindling retail investors out of an undisclosed amount of cash over the past year.
Among others arrested in the massive sting were three brokers at the inter-dealer brokerage ICAP, which operates at a different location, according to another source.
American television reported the defendants scammed retail investors into thinking they were buying multimillion-dollar foreign exchange trades when it was not possible for those types of investors to participate in such deals.
"It's currency fraud... securities fraud. It's been a long investigation. The arrests have been ongoing today"
Unnamed FBI agent
A spokespeople for the US Attorney's office and the Treasury Department has also declined to comment.
The unexpected operation came at a time when America's financial markets have been hit by scandal after scandal.
Two years of scandals
Corporate wrongdoing by companies like Enron Corp, which went bankrupt in 2001, sparked a massive accounting scandal and led to the demise of one of the world's largest accounting firms, Arthur Andersen.
The scandal rocked investor confidence and unearthed irregularities at other companies.
Since then Wall Street equity research companies have been targeted by prosecutors for inflating stocks during the Internet boom of the late 1990s.
Four companies targeted
More recently the mutual fund industry has been investigated on charges of improper trading.
Until Tuesday the $1.2-trillion-a-day foreign exchange market, whose primary clients include top companies, millionaires and banks, has remained relatively untainted by scandal.
The names of other companies involved remained unclear, but it is believed four have been targeted so far.
None of the those targeted are household names outside of the securities industry - but they are well known in the Wall Street community.