The long-running row over the islands, thought to lie near major oil and gas reserves, is one of many issues to strain ties between Japan and China.
According to the Hong Kong-registered China Federation of Defending Diaoyu Islands, Japanese patrol vessels fired water cannon as a boat carrying for of their activists attempted to approach the islands.
The boat had sailed from the Chinese coastal city of Xiamen.
Li Nan, a spokesman for the group, said the protesters had telephoned the group's Beijing office on Sunday evening as they neared the islands.
"At around 7pm, the boat was seven nautical miles [13 kilometres] away from Diaoyu Island," he told the Associated Press.
"The Japanese used water cannon to disperse the boat to around 12 nautical miles [22 kilometres] away. They are now standing off in the waters."
Japanese officials did not confirm that a water cannon had been fired in the incident.
|The islands have been a frequent source of |
tensions between China and Japan [EPA]
Satoshi Matsuyama, a Japanese coast guard official said warnings were issued as soon as the Chinese boat carrying a Chinese flag was first spotted entering the disputed waters near the islands on Sunday evening.
He said coast guard vessels continued to patrol the area after the Chinese boat left the disputed waters an hour later.
The Japanese government has set up a task force to monitor the situation, Japan's Kyodo News agency said.
Japan's foreign ministry said China had responded to the diplomatic protest saying Tokyo's complaint was "not acceptable" and urged Japan to respond calmly.
Japan seized the islands in 1895 when it colonised the island of Taiwan, which also claims the territory as its own.
Disputes over the island flare periodically between China and Japan.
In February Beijing expressed "extreme dissatisfaction" and summoned a Japanese diplomat over Tokyo's complaints about a Chinese research ship that had neared the disputed islands.
In 2004, Japan arrested and deported seven Chinese activists who had landed on one of the islands.