This is according to a controversial new book on the conflict, A French Lie by Georges-Marc Benamou, which forms the basis for a lawsuit alleging crimes against humanity.
The lawsuit is to be filed at the Paris courts on Tuesday, targeting one of de Gaulle's last surviving cabinet members - former armed forces minister Pierre Messmer.
Citing government minutes from 1962, the book claims that in his haste to wrap up the Algerian trauma President de Gaulle pulled out white "pied-noir" settlers, but refused any escape route for the estimated 300,000 "harkis" who faced terrible reprisal for having fought on the French side during the war.
"We cannot move out every Muslim who decides he does not agree with his new government," the book quotes de Gaulle as saying in January, two months before signing of the Evian peace accord with the Algerian National Liberation Front (FLN).
In April that year the president was quoted as telling the cabinet that the "harki" locals were a "rag-tag who served absolutely no purpose and of whom we must rid ourselves as soon as possible," and he was later recorded as exclaiming, "French - those people? With their turbans and djellabas!"
According to Benamou, de Gaulle was partly motivated by his "19th century prejudices ... a frantic desire to keep France white and break with the less ethnic conception prevailing under the Third Republic," which ended with de Gaulle's accession to power in 1958.
The Evian accord stipulated an amnesty by both sides, but within weeks of the withdrawal of French forces later that year the victorious FLN massacred tens of thousands of harkis.
Estimates of those who died vary between 70,000 and 150,000, while only around 40,000 made it to France.
The Charles de Gaulle Institute, which safeguards the late president's memory, reacted angrily to the accusations of racism - drawing attention to "his achievements directed towards the emancipation of overseas peoples," and disputing the "exaggerated" numbers of harkis.
Messmer, who at 87 is one of the few people left alive with inside knowledge of the affair, also denied the harkis had been cynically left to their fate.
Auxilliaries who died are estimated at between 70,000 and 150,000, while only 40,000 made it to France
He told Le Point magazine that they were all offered the choice between integration into the French army or a small pay-off - and that most took the pay-off.
Campaign against mistreatment
Tuesday's law-suit is being lodged by groups representing harkis who have been campaigning for years for recognition of their mistreatment.
In September 2001 they won the first acknowledgement of official French shame with a "day of national homage" in which President Jacques Chirac unveiled a plaque in their memory.
"It was a genuine act of ethnic cleansing"
Their lawyer Emmanuel Altit said with the passage of time new evidence had come to light which proved their case.
"It was a genuine act of ethnic cleansing. Startling new documents are appearing from the period and people want to speak out. The aim of our plea is judicial ... but it is also an act of education and of therapy. France's perception of its victims has changed," he said.