The survey, conducted by the Bank of Japan (BoJ) and released on Tuesday, showed that the index for large manufacturers' business sentiment fell to five in June, down from 11 in the March survey.
The figure represents the percentage of companies saying business conditions are good minus those saying conditions are unfavourable.
The survey covers more than 10,000 companies.
The latest result was the lowest in almost five years but slightly better than market predictions for a reading of between two and four.
The decline in confidence, however, was broad-based and hit small- and medium-size firms especially hard.
The index for small manufacturers tumbled to minus 10, while for small non-manufacturers its dived to minus 20.
The survey results did not appear to rattle investors with the benchmark Nikkei-225 share index rising slightly, nudging up 0.33 percent by midday on Tuesday.
Japanese business confidence has fallen sharply from a two-year high of 25 seen in the tankan survey in December 2006, but it is still much higher than a low of minus 38 hit six years ago.