"Sponsors and individual athletes make their own decision on whether to [boycott] the games"
David Kilgour, veteran Canadian politician
In advising the Chinese Taipei Olympic Committee, Gosper said it "should stick by their agreement".
"They should recognise that they have a special status in the IOC and get on with it or get out," he was quoted as saying in The Australian.
Gosper said he did not know if the issue could be resolved but made it clear which party the IOC would be backing on the torch relay row.
"When we recognised the PRC as the national Olympic entity in the late 70s, a solution was found for Taiwan," he said.
"It was a unique outcome and the mainland Chinese agreed with it. It was a very good settlement," added Gosper.
Taiwan split from mainland China at the end of the Chinese civil war in 1949
Beijing regards Taiwan as part of its territory, and says it will use force if it ever declares independence
Beijing and Taipei both say they are the legitimate government of all China
Taiwan has been a multi-party democracy since 1996
Taiwan's defence ministry says China now has nearly 1,000 missiles aimed at the island
The US is the major arms supplier to Taiwan and has warned China that any attack on the island would be viewed with "grave concern"
Meanwhile, human rights activists launched a torch relay of their own to highlight what they say are widespread abuses in China.
The event in Athens, Greece, the birthplace of the Olympics, was held on Thursday as China marked the one-year countdown to the Summer Olympics.
The protest, led by followers of the Falun Gong spiritual group which China has banned, attended a flame-lighting ceremony in central Athens for the relay, which protest organisers said will include about 25 countries.
Greece is the birthplace of the Olympics and hosts the official flame lighting ceremony at Ancient Olympia for all Summer and Winter Games.
About 300 people mostly from Greece and China attended the small ceremony at the city's main Syntagma Square, a common focal point of protest rallies.
David Kilgour, a former secretary of state for Asia-Pacific, said human rights was "getting steadily worse in China over the last couple of years".
"Sponsors and individual athletes make their own decision on whether to [boycott] the games. I don't think governments will do it, for all the wrong reasons," said the veteran Canadian politician who helped organise the event.
The Beijing Olympics torch relay would include most major Canadian cities and several in the US, as well as European and Asian countries.
Jan Becker, a former Australian Olympic swimmer who was also visiting Athens, said: "Human dignity is more important than any sporting event."