He said: "I believe that from the standpoint of how it reflects on us that it's been pretty damaging."
But he acknowledged closing the prison posed major legal problems.
"There are enormous challenges associated with that," Mullen said.
"There are enormously complex, complicating legal issues that are way out of my purview."
He said there were some "really, really bad people" detained at Guantanamo who had committed "extraordinary crimes".
Among those detained is Khalid Sheikh Mohammed, the self-confessed mastermind of the September 11, 2001, attacks in New York and Washington DC.
Mullen also said steps had been taken to reduce the population at the camp, which now stood at 277.
Earlier, Mike McConnell, the chief of US national intelligence, was quoted as saying he believed the US interrogation practice known as "waterboarding" could be described as torture.
"Whether it's torture by anybody else's definition, for me it would be torture," he told The New Yorker magazine on Sunday.
|Protesters enact 'waterboarding' in one of the|
many rallies held against its practice [Reuters]
The comments came as the US House Intelligence Committee continues an investigation into the CIA's destruction of videotapes that are reported to have shown the use of the interrogation technique on suspects.
McConnel also said that should waterboarding ever be determined as torture, "there will be a huge penalty to be paid for anyone engaging in it".
"If I had water draining into my nose, oh God, I just can't imagine how painful!" he said.
"Waterboarding", involves pouring water over subjects who are bound, gagged and hooded in order to terrify them by stimulating the feeling that they are drowning.
McConnell stopped short of categorically describing the interrogation process as torture and declined to say whether he believed it should be formally labelled as such.
A spokesman for McConnell later said he did not dispute the quotes attributed to him in the story the Associated Press reported.
Kevin Lanigan, from Human Rights First, told Al Jazeera: "It's a very important step for such a senior official in this [US] administration for the first time to admit that - with some caveats on his admission - that this technique is torture."
Michael Mukasey, the US attorney general, has declined to rule on whether "waterboarding" is torture.
A ruling that the technique does constitute torture would put at risk the CIA interrogators who were given permission by the White House in 2002 to use the technique on three prisoners who were considered resistant to conventional interrogation.
McConnell said in his interview that the legal test for torture should be "pretty simple", suggesting: "Is it excruciatingly painful to the point of forcing someone to say something because of the pain?"
The CIA has said it has not used the technique since 2003 and Michael Hayden, the agency's director, prohibited it in 2005.
Tony Fratto, a White House spokesman, refused to comment on the issue on Saturday.
He said: "We don't talk about interrogation techniques. And we are not going to respond to every little thing that shows up in the press.
"We think it's vitally important he and the intelligence community have all the tools they need."
Source: Al Jazeera and agencies