General Ilker Basbug, the head of Turkey's armed forces, has postponed a trip to Egypt as a result of the investigation.
The AFP news agency reported that seventeen retired generals, including Ibrahim Firtina, the former head of the airforce and Ozden Ornek, a former navy chief, as well as four active-duty admirals were among those detained.
The operation has brought condemnation by opposition figures.
"This is not a legal process. This is apparently a sheer process of political showdown," said Deniz Baykal, head of the main opposition Republican People's Party.
Devlet Bahceli, another opposition leader and head of the Nationalist Action Party, accused the government of carrying out a "political vendetta" and urged the sides to refrain from making inflammatory statements that could further fuel the tensions.
The arrests are believed to be linked to a purported 2003 plot to discredit the AKP government, a plan revealed by the Taraf newspaper in January.
The plan, codenamed Sledgehammer, involved bombing two Istanbul mosques and escalating tensions with Greece by forcing Greek jets to down a Turkish plane over the Aegean Sea.
The act would have been an attempt to show the Turkish government as inept, according to documents obtained by Taraf.
The Turkish army said the documents were discussed in a seminar on war-time contingency plans, but denied they represented a coup plot.
Anita McNaught, Al Jazeera's correspondent in Istanbul, said: "When the head of the military last spoke to the public on the subject he appealed to their common sense.
"He said how could army that chants 'Allah' when it orders its troops into battle be the same army that could possibly conceive of bombing a mosque and injuring Muslims at prayer.
"But it seems that those allegations may have foundations and that's what the investigations are about.
"And we're expecting some of them [those detained] to come to some sort of court hearing this week."
The armed forces have ousted four governments since 1960 and such an operation against them would have been unimaginable until recent years.
Under EU pressure, however, Erdogan has dramatically curtailed the military's power and reinforced its place under civilian rule.
Other senior military officers have been indicted on charges of planning a separate plot by a far-right group known as "Ergenekon" to overthrow the government.
That trial is ongoing. Critics of the government say the Ergenekon case has been used to target political opponents.
The Turkish lira and shares weakened in early trading as the investigation hurt investor sentiment.