Relatives said that although physically weak, he remained completely lucid and enjoyed speaking with many of his friends worldwide over the past few weeks.
Although not widely known to the jazz public until late in an extraordinarily long career, fellow musicians nicknamed him "the King" for his astonishing versatility, and critics recognized him as a major influence in American music.
Nominated for seven Grammys, he won two as well as a Grammy Lifetime Achievement Award in 1987.
In 1990 he was named "Jazz Artist of the Year" in Down Beat and Jazz Times International Critics' polls.
Carter is considered among the greatest jazz alto saxists of the swing era.
Self-taught in Harlem
Born in New York in 1907, Carter was sitting in at Harlem night spots by the age of 15, although he was largely self-taught, according to a biography.
But he was far more than a giant of jazz saxophone, he was at one time or other a composer and arranger, a skilled clarinetist and gifted trumpet player and an exceptionally versatile, talented musician.
His signature sound was smooth, flowing and graceful - displaying ease, confidence and a clear sense of direction when soloing.
Making written arrangements sound clean and spontaneous, his written arrangements in the 1920s and ‘30s helped create the basic building blocks of Big Band Swing for Fletcher Henderson and Benny Goodman among others.
Carter was a very good clarinetist in his early years, an excellent jazz trumpeter (he influenced no less than Dizzy Gillespie) and blessed with perfect pitch. More amazingly, he was largely self-taught.
Later, he composed for Hollywood films and television, and taught at Princeton University where he was awarded a Doctor of Humanities in 1974.
Benny Carter remained very active for many years. He performed for at least three US Presidents and received numerous awards including a National Medal of Arts in December 2000.
His compositions and arrangements for other artists read like a Who's Who of jazz instrumentalists and singers, including Count Basie, Duke Ellington, Glenn Miller, Tommy Dorsey, Billie Holiday, Ella Fitzgerald, Sarah Vaughan, Lou Rawls, Ray Charles, Louis Armstrong and Mel Torme.
He helped shape big band and swing jazz as a bandleader from 1929 to 1946, while work overseas in the 1930s helped spread jazz in Europe.
Carter toured Europe from 1935 to 1938, playing mostly in France, Denmark and the Netherlands, and spent 10 months as arranger for the BBC dance orchestra.
Younger musicians like Miles Davis and Max Roach played in his bands and considered him a teacher.
"The problem of expressing the contributions that Benny Carter has made to popular music is so tremendous it completely fazes me, so extraordinary a musician is he," Duke Ellington once wrote.