Lord Peter Goldsmith said in a speech in London on Wednesday that "the existence of Guantanamo Bay remains unacceptable", adding that "it is time, in my view, that it should close".
Goldsmith, the highest British official to make an unambiguous call for the prison's closure, said shutting it down would help the US burnish its global image.
"Not only would it, in my personal opinion, be right to close Guantanamo as a matter of principle, I believe it would also help to remove what has become a symbol to many - right or wrong - of injustice," he said.
"The historic tradition of the United States as a beacon of freedom, liberty and of justice deserves the removal of this symbol," he told a conference on terrorism at the Royal United Services Institute think-tank.
Goldsmith, responsible for advising the government on legal matters, was involved in negotiations with the US government over Britons held in the Guantanamo camp in Cuba which is run by the US Navy.
Kate Allen, who heads the British branch of the London-based human rights group Amnesty International, welcomed Goldsmith's call but said more needed to be done.
"The real test is whether the UK government is prepared to put serious pressure on the US administration to see that the camp is closed and that all prisoners are released to safe countries or brought before proper courts on the US mainland," Allen said.
"The real test is whether the UK government is prepared to put serious pressure on the US administration to see that the camp is closed"
For four years, some 500 detainees have been held at Guantanamo but only 10 have been formally charged as "terrorists".
Most were captured after the US-led war in Afghanistan against the Taliban government in the wake of the September 11, 2001 attacks in the United States.