The network, which will run 508km between Mumbai and Ahmedabad, was announced on Thursday by Prime Minister Narendra Modi.
"This is the new India, and the flight of its dreams is endless," he said. "The bullet train project will bring speed and employment. It is human-friendly and eco-friendly."
Japan will fund more than 80 percent, nearly $1.4bn, of the project's cost, providing a 0.1 percent interest loan due to be repaid over the next 50 years.
Shinzo Abe, the Japanese prime minister, said the development - a joint venture between Indian Railways and Japan's Shinkansen Technology - will prove beneficial for both nations.
"A strong India is good for Japan, and a strong Japan is good for India … We are fully committed to supporting this initiative," he said on Thursday.
Abe said the deal marks increasing levels of cooperation between the two regional powers.
"The Indo-Japan relationship is not just about bilateral trade. It has developed into a strategic and global partnership in the Indian-Pacific region," he said.
The line, which will use Japanese bullet trains and technology, is expected to be complete by 2023 and will cut travel time between the two cities from eight hours currently to around three.
The new train will carry 750 passengers at an average speed of 250 kilometres per hour and a top speed of about 320 km/h, double the top speed of the fastest train in India.
Modi has promised to invest billions of dollars to modernise India's railway system, which carries more than 22 million passengers daily.
On Thursday, the two sides also signed agreements covering science and technology, investment promotion, disaster management and civil aviation cooperation after the discussions.
The Japanese leader's visit and the launch of the projects are seen as symbolic of the growing ties between New Delhi and Tokyo amid China's growing assertiveness in the region.
Source: News agencies