Eliot Spitzer said his legal action would address the issue of Richard Grasso's $188.5 million pay package, which he described as "rigged" and "wholly inappropriate and illegal".
Spitzer told a press conference he would be seeking the return of "at least" $100m from Grasso and aimed to have a judge rescind the pay deal and determine a "reasonable" level of compensation.
The action is being taken under state not-for-profit laws.
After a four-month inquiry, the attorney general added that the former NYSE chairman had misled and intimidated some exchange board members.
Grasso stepped down from the stock exchange in September 2003 when a controversy surrounding his pay deal was at its height.