The Australian Strategic Policy Institute said China's ageing communist leadership was committed to peacefully pursuing economic expansion, but that could change when the next generation of leaders takes over.
In a report entitled In the balance: China's Unprecedented Growth and Implications for the Asia-Pacific, the institute said countries in the region must carefully engage China.
It said the impending generational change in leadership meant Tokyo and Washington could not afford to isolate China because of their concerns about its increasing economic power and burgeoning defence spending.
The report said: "When a fifth generation of leadership assumes power in 10 to 15 years, China could become more open and tolerate greater dissent.
"Such a political opening could then open the door to forces such as nationalism and populism. There is no way to predict exactly how Chinese politics will evolve in a more democratic era, but it is a development which could produce new challenges for the countries of East Asia after 2020.
"An authoritarian China has been highly predictable. A more open and democratic China could produce new uncertainties about both domestic policy and international relations."
The report's author, David Hale, a US-based economist and China specialist, said Taiwan and North Korea represented the major potential flashpoints involving China, along with Beijing's long-standing differences with Japan.
But the report said the current Chinese leadership had a vested interest in maintaining stability in the region because exports underpinned its booming economy.
The report adds: "It is unusual for a country as large as China to be so heavily dependent upon foreign trade, but as a result of low labour costs, good infrastructure, and pro-business economic policies the global corporate community has turned China into a workshop of the world.
"China has become so integrated with the global economy that she can no longer pursue a high-risk foreign policy without jeopardising her economic prosperity."
Hale said China was likely to threaten other countries only if there was domestic political instability which produced an upsurge in nationalism and a search for external scapegoats.
He said such instability could be sparked if dissatisfaction over increasing income inequality in China resulted in a populist government that suspended economic reforms.