Oscar Arias, a former Nobel Peace Prize winner, said on Wednesday that he made the decision to win more friends in the Middle East and comply with United Nations' resolutions.
"It's time to rectify an historic error that hurts us internationally and deprives us of almost any form of friendship with the Arab world, and more broadly with Islamic civilisation, to which a sixth of humanity belongs," Arias said at an event marking his first 100 days in office.
He said Shimon Peres, Israel's deputy prime minister, had called him on Tuesday to ask him to reconsider the decision.
Israel regards East Jerusalem as part of its "undivided and eternal capital". It captured the eastern part of Jerusalem in the 1967 Middle East war and annexed it in a move not recognised internationally.
The 200,000 Arab residents of East Jerusalem are caught between Israel and an emerging Palestinian state, which wants the eastern part of Jerusalem as its future capital.
Luis Alberto Monge, the former Costa Rican president, moved the embassy to Jerusalem in 1982 as a show of support for Israel.
Arias said the relocation to Tel Aviv should not, however, be interpreted as a slight to Israel, which has had historically close ties to Costa Rica.
"As far as Costa Rica is concerned, the right to exist and live free of threat, particularly the criminal threat of terrorism, is beyond all doubt," said Arias, who won the Nobel Peace Prize in 1987 for his efforts to end civil wars in neighbouring Central American nations.
The decision will leave El Salvador as the only country in the world with an embassy in Jerusalem.