Qatar News Agency said on Tuesday that Sheikh Hamad bin Khalifa Al-Thani accepted the resignation of Sheikh Jassim bin Hamad bin Khalifa al-Thani on Tuesday.
The emir's fourth son, Sheikh Tamim bin Hamad bin Khalifa al-Thani, was named as the new crown prince.
Sheikh Tamim, who heads Qatar's Olympic committee, is in his 20s.
The emir offered no immediate explanation for the sudden change, but diplomats say that Sheikh Jassim has complained of being sidelined in the running of the oil-and-gas rich country.
Qatar News Agency said Sheikh Jassim sent a letter to his father saying "the time is appropriate to step down and prepare for a successor".
Qatar is rich in oil and gas
The letter did not give specific reasons but Sheikh Jassim said he first accepted the post in 1996 to help his country after authorities foiled a coup against his father.
But diplomats told Reuters that Sheikh Jassim was frustrated because he did not have enough official responsibilities.
Another diplomat said the change might be due to health reasons, but did not elaborate.
Qatar, a small state with a history of palace coups, was the springboard US forces used to launch their war against Iraq in April.
The OPEC member has the world's third largest natural gas reserves.
Sheikh Hamad, 54, ousted his father in 1995 in a bloodless palace coup while the old emir was in Switzerland.