The company said that O'Neal "decided to retire" immediately and the board of directors have elected Alberto Cribiore, another board member, as an interim non-executive chairman.
'Change in leadership'
"Mr O'Neal and the board of directors both agreed that a change in leadership would best enable Merrill Lynch to move forward and focus on maintaining the strong operating performance of its businesses," Merrill Lynch said in a statement.
The firm also said Cribiore would chair a search committee to recruit a new CEO from within or outside its ranks to oversee an estimated $1.7 trillion in client assets.
Cribiore is a managing partner and founder of Brera Capital, a global private equity firm.
He said that Ahmass Fakahany and Gregory Fleming will continue as Merrill Lynch co-presidents and chief operating officers.
O'Neal's departure came after Merrill said last Wednesday that it was writing off $7.9bn of mainly mortgage-backed securities, resulting in a third-quarter operating loss of $2.24bn.
The earnings news startled investors after Merrill forecast on October 5 it would take a much lower $4.5 billion writedown.
O'Neal at the time admitted: "We can do a better job in managing this risk."
With his exit, Wall Street loses the only African-American head of a major investment bank, a man who rose from poverty in the south of the US to the top echelons of corporate America.
He said: "I have been very fortunate to spend the past 21 years at Merrill Lynch.
"The company has provided me with opportunities that I never could have imagined growing up, culminating with my leadership of the company over the past five years."
John Challenger, chief executive of Challenger, Gray & Christmas, a global outplacement consultancy, said O'Neal's departure could will not be the last from a powerful figure amid the country's financial woes.