Cabinet reshuffle follows summer defeat by Russia over breakaway regions.
Utiashvili said that between 500 and 600 Russian troops moved into Perevi.
On Friday, Georgian and EU monitors had announced that Russian forces were withdrawing from Perevi, a mainly ethnic Georgian village with a population of around 1,100 on the western border of South Ossetia.
It has been under Russian control since a five-day war in August that followed months of skirmishes between Ossetian separatists and Georgian troops. Pro-Russian South Ossetia threw off Tbilisi rule in 1991-92.
“The renewed Russian military occupation of the Perevi checkpoint … is incompatible with the … peace plan”
EU mission statement
The move has drawn sharp criticism from the European Union (EU) monitoring mission that is in Georgia to observe the EU-brokered ceasefire.
The 225-member EU mission issued a statement calling on Russia to pull out of the village.
“The renewed Russian military occupation of the Perevi checkpoint, and in addition, the Perevi village, is incompatible with the provisions of the … peace plan,” the statement said.
The mission also complained that a delegation of European ambassadors visiting the area were prevented from entering the village, saying the refusal was “unacceptable under all relevant instruments of international law”.
John Kerry, US senator and former presidential candidate, who was in Tbilisi on a half-day visit on Saturday, said: “My judgment is that Georgia as a sovereign country needs to be upheld and respected … and the agreement that the Russians have signed up to needs to be upheld.”
Georgia has also called for Russia to withdraw from the Akhalgori district in South Ossetia and the Kodori Gorge in Georgia’s other separatist region, Abkhazia. Both areas were under Tbilisi’s control until the conflict in August.
Russian forces moved into Georgia in August in response to a Georgian military attempt to retake South Ossetia.
Russia later withdrew to within South Ossetia and Abkhazia, which Moscow recognises as independent states.
The Russian foreign ministry refused to comment immediately and South Ossetian officials could not be immediately reached, news agency AP reported.