The association of Japan's North Korean residents, known as Chongryon, said it had called the rally to demand an end to police harrassment and the bullying of North Korean schoolchildren, which it says have intensified since Pyongyang carried out missile tests in July.
The protesters also want the resumption of a ferry between the two countries, which Japan shut down permanently after North Korea tested an atom bomb in October.
Chongryon estimated about 3,000 people attended the rally.
"Japan's violations of human rights against the DPRK [Democratic People's Republic of Korea] and Koreans in Japan cannot be allowed, no matter what," said Nam Sung-U, Chongryon's second-in-command.
"We Koreans in Japan have gone through such suffering during colonial rule and even after liberation. We have united to survive," he said.
About 700,000 Koreans live in Japan, mostly a legacy of those who emigrated or were enslaved during colonial rule. A disputed number are affiliated with North Korea.
Shinzo Abe, the Japanese prime minister, built his career on campaigning against North Korea, particularly over its abductions of Japanese nationals in the 1970s and 1980s.