The move is likely to provoke an outcry from China and other nations who suffered from Japan's war-time aggression.
It also comes as Tokyo is in the process of sending military personnel to Iraq, despite fierce opposition at home.
Since taking office early in 2001, the strongly nationalistic Koizumi had already outraged Asian neighbours three times with visits to the Yasukuni Shrine in Tokyo, seen by many as a symbol of the militarist regime that led Japan into war.
The shrine is dedicated to Japan's 2.5 million military war dead since 1853, including wartime prime minister Hideki Tojo and 13 other convicted "Class A" war criminals.
Koizumi last went to the shrine in January 2003, but has never before visited on New Year's Day, traditionally a time for Japanese to go to their local shrines and pray for a good year.
A solemn Koizumi, dressed in a formal kimono and a black haori coat, bowed briefly as he entered the shrine. "I feel refreshed," he told reporters afterwards.
"One does not comment about another country's respect of its history, traditions or customs"
Japanese Prime Minister
Asked if he worried about the response from China and North Korea, wartime victims of Japanese aggression, the prime minister replied: "One does not comment about another country's respect of its history, traditions or customs."
He added he believed those countries would come to understand the reasons why he visited the shrine.
The prime minister waved at holiday crowds as he got back into his car. They shouted back "Happy New Year!"
Huang Xingyuan, spokesman of the Chinese Embassy in Tokyo, was quoted as expressing strong dissatisfaction over the visit.
The latest visit comes at a delicate time as diplomats from six nations, including Japan and the United States, are working to arrange a second round of six-party talks on North Korea's nuclear weapons programme.
North Korea, along with China and South Korea, has lambasted Koizumi for his past visits to the shrine.
Koizumi has avoided visiting the shrine on 15 August, the anniversary of Japan's 1945 surrender and an emotive date in the region, but the visits have nonetheless infuriated China and other Asian neighbours.
War veterans are a key part of Koizumi's support base, essential as his ruling Liberal Democratic Party gears up for an election in July for parliament's Upper House.
This year's shrine visit also comes as Koizumi prepares to send troops to Iraq for reconstruction of that war-torn nation, a controversial mission over which the public is deeply divided.