Summers sparked a furore last month when he theorised that "intrinsic aptitudes" may explain why more men work in the academic sciences than women.
Although he said he was intentionally trying to provoke and that he wanted to be proven wrong, his comments brought calls for his resignation from faculty and students who accused him of insensitivity.
"Larry Summers is Exhibit A in how you can step into the culture of offence in American academic life," said David French of the Foundation for Individual Rights in Education.
French said the dust-up over Summers' remarks had its roots in the late 1980s and early 1990s, when US universities adopted laws aimed at prohibiting speech that would be offensive on the basis of race, gender and sexual orientation.
"The paramount right at US colleges has become the right not to be offended rather than the right to challenge convention and create a free marketplace of ideas," he said.
"The paramount right at US colleges has become the right not to be offended rather than the right to challenge convention and create a free marketplace of ideas"
David French, Foundation for Individual Rights in Education
Summers has repeatedly apologised for his remarks, said he was trying to foster debate and has pledged to temper his words. But some of his faculty critics are not satisfied.
Professor Lorand Matory plans to call for a vote of no confidence in Summers at next month's undergraduate faculty meeting, arguing that the Ivy League school chief should not have said what he said because of his position.
"This is the president of the most prestigious university in the nation, and whether he realises it or not ... what he says is likely to be taken as the official university position," Matory said.
Professor Susan Suleiman agreed, saying: "He really failed to understand what effect the words of the president of Harvard can have."
The controversy over Summers has polarised the stately Harvard campus but also created strange bedfellows.
Several dozen students waved fluorescent signs and chanted anti-Summers slogans at a campus rally earlier this week. But off to the side stood a smaller group of Summers' supporters.
Among them was Noah Hertz-Bunzl, a scruffy-looking freshman from New Jersey with a sign reading: "I support Pres Summers".
Hertz-Bunzl said he favoured women's rights and opposed sexism, but was even more opposed to an academic environment that stifled discussion and debate.
"It's troubling when members of the faculty feel it necessary to silence a point of view with which they disagree"
"To limit free speech goes against what a university stands for," the Democrat-supporting student said.
The sentiment was echoed by Matthew Downer, a clean-cut sophomore and a Republican supporter.
"It's troubling when members of the faculty feel it necessary to silence a point of view with which they disagree," Downer said. "What we've seen over the last month is the slow undoing of those academic freedoms to which Harvard has always been committed."
Harvard is not alone as it argues such issues.
At Columbia University in New York, pro-Israeli students have complained about harassment by faculty in the school's department of Middle East and Asian Languages and Cultures.
Elsewhere, a little-known University of Colorado professor, Ward Churchill, has been condemned by conservatives for what they see as his use of offensive language.
In an essay penned shortly after 9/11 and which came to the media's attention in January, Churchill said victims of the World Trade Centre attack could not be considered innocent and called them "little Eichmanns" - a reference to Nazi war criminal Adolf Eichmann.
Churchill has conceded his analogy was "an ill-advised rhetorical device", but insists on his point that the attacks were understandable given hostile US policy in the Middle East.
Churchill has become fodder for talk show hosts and editorial writers nationwide.
The real issue, Churchill said, is academic freedom. He said a "fascist bloc" of right-wing politicians and talk show pundits were out to stifle not just dissent, but any disagreement with US foreign policy.
"They have made me a target and I'm going to shoulder that role," he said.