The submarine was detected on Wednesday in Japanese waters near islands disputed with China about 300km south-west of Okinawa, a southern Japanese island home to a major US military base, officials in Tokyo said.
Japan was following the submarine with a PC-3 surveillance airplane, a destroyer and at least one navy helicopter, a military spokesman said on Wednesday.
Prime Minister Junichiro Koizumi said the incident was "regrettable."
"It certainly is not a good thing," Koizumi told reporters. "We have to continue to monitor the situation."
Japanese used PC-3 surveillance
aircraft to follow the submarine
A Defence Agency spokesman said Japan had spotted between Friday and Monday two Chinese ships near southern Japan - one designed to rescue submarines and the other to tow wrecked ships.
The Chinese ships were in international waters about 1000 km south of Tokyo, the spokesman said.
Asked if the Chinese ships had a link with Wednesday's submarine incident, the spokesman said: "We don't know."
Japan wants the submarine to surface and show its flag, but has not given orders to attack the vessel as it is now in international waters, the Defence Agency spokesman said.
The submarine was in Japanese waters for about three hours.
Hosoda said the submarine was found near Okinawa's Sakishima island chain, which lies close to islands disputed between China, Japan and Taiwan - known as Senkaku in Japanese and Diaoyu in Chinese.
"It certainly is not a good thing. We have to continue to monitor
Japanese Prime Minister
China feels deep resentment over Japan's alleged refusal to admit its atrocities during its occupation from 1931 to 1945.
A Japanese study has meanwhile said China was trying to strengthen its military power to "demonstrate its capability to Taiwan and the United States and will be the greatest military power in the Asia-Pacific region in the future."
The report envisioned three scenarios for a Chinese attack of Japan, including one focused on Okinawa in which China would try to stop US forces from helping Taiwan in the event of a conflict with the island, which China regards as its territory.
Chinese foreign ministry spokeswoman Zhang Qiyue Zhang on Tuesday accused Japan of a "Cold War mentality" and said it was not certain that President Hu Jintao would meet Koizumi when they attended a regional meeting in Chile later in November.