Father Christian Bouchacourt, the head of the South American chapter of the society, said: "It is clear that a Catholic bishop cannot speak with ecclesiastical authority except on matters concerning faith and morality."
Bouchacourt described Williamson's remarks as "inopportune" and said he reacted to them "with sadness".
Pope in talks
The Society of St Pius X was founded by Marcel Lefebvre, a French archbishop, in 1970 as a protest against the who did not accept the Vatican's reforms of the early 1960s.
Williamson and three other bishops were excommunicated 20 years ago after they were consecrated by the Lefebvre without consent from the then pope, John Paul II - a move the Vatican said at the time was an act of schism.
Pope Benedict XVI sparked anger last month when he lifted excommunications on Williamson and the three other bishops, in an attempt to heal the rift within the church.
The pope has said he was not aware of Williamson's denial of the Holocaust when he lifted the excommunication orders.
On Sunday, Angela Merkel, the German chancellor, and the pope had held talks on over his decision to lift the excommunications.
An official statement said they had a "cordial and constructive" conversation.
It was "marked by the common and profound understanding that the Holocaust still represents a valid warning to humanity," the statement said.
Last week, Merkel had criticised the pope for rehabilitating the four bishops.