According to the Vatican statement, Glemp will temporarily take back the position.
Al Jazeera's correspondent in Warsaw said there were scuffles outside St John's cathedral - where the inauguration was to be held - between supporters of the archbishop and those calling for his resignation.
Wielgus had resisted calls to step down after allegations that he collaborated with the secret services during the communist era.
The archbishop backed down last Friday and admitted to collaborating with the communist regime.
The Polish Episcopate could not be reached for comment.
The Sunday service was going ahead as planned, our correspondent said, and would be held as a thanksgiving to Cardinal Jozef Glemp.
In a statement issued on Friday, Wielgus appeared to back down from earlier denials in the row and to open the door for the pope to remove him from office.
On the church's website, Wielgus said in a statement: "By the fact of this entanglement I have damaged the church ... I will respect any decision the pope makes.
"I damaged the church again when in recent days, amid a hot media campaign, I denied the facts of this co-operation."
Soon after his appointment, Polish media reported Wielgus had informed on fellow clerics for about 20 years from the late 1960s.
Wielgus maintained in Friday's statement that he "did not report on anyone nor deliberately try to hurt anyone".
But a special church commission said in a statement there was sufficient evidence to confirm that he was a willing informer.
Wielgus formally took up his job on Friday and was due to be ceremonially invested at a service on Sunday attended by the president and other government officials.