Officials said Costas Kartalis resigned despite pleas from new Prime Minister Costas Karamanlis to remain in his post for the 13-29 August games.
In charge of day-to-day operations, Kartalis had only been subordinate to Olympics chief Gianna Angelopoulos and the country's Culture Minister.
But he decided to leave due to unease about working for the new government, formed by the conservative New Democracy party.
Kartalis took over as general secretary in 1999 when the then ruling socialists were struggling to speed up preparations for the Olympics after losing two years in political infighting.
Karamanlis, who swept to power last Sunday, had made retaining top Olympic officials one of his first promises during the election.
In a sign that he is now personally in charge of the troubled Games, Karamanlis has taken over the Culture Ministry - which supervises Olympic preparations.
Of the 39 Olympic venues, only 24 have been fully completed and many transport infrastructure projects are also behind schedule.
International Olympic Committee president Jacques Rogge will meet Karamanlis in Athens on Saturday to discuss the new government's plans to speed up preparations.