News of the deal, first reported by the Wall Street Journal, propelled Google shares to a record high.
Under the terms of the proposed deal, which could close within days, Google would continue to provide AOL with its paid search advertising technology for five years, the source said.
The deal would allow AOL to sell display and banner advertising to other websites using Google's ad serving technology, the source said.
AOL accounts for from 2% to 4% of Google's net revenue, analysts have said.
Microsoft had been negotiating to get AOL to use its search technology instead, which would have almost immediately given the software giant a huge presence in its fledgling paid search business.
Richard Greenfield, an independent analyst, said: "AOL would have been a huge opportunity and a natural fit to jump-start that business for Microsoft.
"If Google has boxed Microsoft out, it begs the question: How is Microsoft going to flourish online?"
Neither Google, Microsoft nor Time Warner spokesmen would comment.
Under the terms of the discussions, AOL's programming will be promoted through Google's paid search listings in a special section of Google's "sponsored search listings" that are often at the top of its search results, the source said.
Microsoft proposed a joint venture with AOL to combine the two companies' advertising businesses that would have relied on Microsoft's advertising system to run ads across AOL's network of sites, said another source familiar with the matter.
"AOL would have been a huge opportunity and a natural fit to jump start that business for Microsoft"
Richard Greenfield, independent analyst
The proposal at the end of the negotiations did not involve a possible equity for Microsoft in AOL, the source said.
"There was always an elephant whose name was Carl," the source said, speculating that a joint venture with Microsoft would make a possible AOL spin-off difficult.
Google's investment gives AOL a value of $20 billion.
Google's re-emergence as front-runner for the AOL search and advertising business will, if a follow-on deal is consummated, mark the second time that Google has wrested a contract with AOL away from its rivals.
A year ago, AOL Europe had awarded its advertising business to Yahoo! but after a last-minute intervention by Google executives was able to secure the AOL Europe advertising contract, according to published reports.