Some opposition politicians also accuse Brown of trying to avoid angering China, which routinely protests when foreign politicians meet the Dalai Lama.
However, the Dalai Lama said that his visit was "basically non-political". He also said that he was encouraged that Hu Jintao, the Chinese president, had publicly acknowledged a meeting that took place between representatives of Tibet's government-in-exile and Chinese officials on May 4.
It was the first such meeting since 2006.
He said: "The president himself acknowledged that meeting, and he expressed seriousness about that meeting.
"That never happened in the past."
Hu said earlier this month that he was willing to continue the recently opened dialogue with the Dalai Lama, although he accused the Tibetan leader of inciting violence and attempting to undermine the upcoming Olympic Games in Beijing.
The 72-year-old spiritual leader said time would tell whether China's attitude had changed, or whether authorities were seeking to avoid bad publicity ahead of the Olympics.
He said: "After the next meeting and also after the Olympics ... I think we can see whether the Chinese government's desire to meet our people was only for the Olympics or more serious."
China drew international criticism for its violent response to protests against Chinese rule that began in March in Lhasa, the Tibetan capital, and soon spread to neighbouring regions.
In the wake of the protests, demonstrators targeted the international Olympic torch relay at stops in Europe and North America.
China has ruled Tibet since it invaded in 1950, and claims the Himalayan region as a part of its territory.
The Dalai Lama fled into exile in 1959 after a failed uprising against Chinese rule.
He says he supports autonomy for the predominantly Buddhist region, but not independence.
He denies involvement in the spring protests, but acknowledged they reflect some Tibetans' frustration with his "middle-way" approach, which has yielded few concrete results after almost half a century.
But he insisted Tibet's future lies "within the constitutional framework of China", and said China's rise as a global economic and political power would benefit Tibet.
Tibet, he said, was "spiritually advanced" but "materially backward". "Every Tibetan wants to modernise Tibet ... they want more prosperity.
They want material development. For that reason, it is in our interests to remain within the People's Republic of China."
The Dalai Lama called on Tibetans not to disrupt the Olympic torch relay, and said he would consider attending the games if political dialogue with China went well.