Kosumi was a prominent member of the Kosovan team negotiating the province's future in ongoing UN-sponsored talks.
He announced the resignation on Wednesday after presenting it to his cabinet, while the chief UN envoy for the status talks was negotiating the future of the province just one floor below.
"I consider the resignation as a right and a moral act," Kosumi told reporters.
Kosumi's resignation is part of a broader governing coalition deal reshuffle. Another key figure, parliament head Nexhat Daci, will also be replaced, senior officials from the government's two coalition partners told The Associated Press.
However, it remained unclear whether Daci would voluntarily step down, after he was voted out by his party.
Kosovo's government is made up of the Democratic League of Kosovo and the minor Alliance for the Future of Kosovo.
Kosumi's popularity has dropped sharply in Kosovo due to a perception that he is politically weak. He became prime minister following the resignation of Kosovo's former prime minister, Ramush Haradinaj, who was indicted last year for war crimes by the UN war crimes tribunal in The Hague.
The resignation plunges Kosovo in further crisis as talks on its future settlement have been launched. Kosovo's ethnic Albanians are demanding full independence from Serbia. Serbian leaders in Belgrade seek to maintain at least some control over the province.
Kosovo has been a de-facto UN protectorate since the end of the war there in 1999.
Among those mentioned to replace Kosumi is Lieutenant General Agim Ceku, the former rebel commander of the now-disbanded Kosovo Liberation Army and the head of Kosovo Protection Corps, a civil emergency organisation, which ethnic Albanians view as the nucleus of their future army.