Sarkosian's closest rival, Levon Ter-Petrosyan, criticised the OSCE's comments and described the election as "shameful".
He told the crowd "we have a precise plan and we will win".
The protesters congregated on the central Freedom Square on Wednesday after marching through the city.
Government buildings and particularly the election commission were guarded by large numbers of riot police.
Despite some angry exchanges and shaking of fists, the demonstration remained calm.
Favoured candidate
Sarkisian was the favoured candidate of Robert Kocharian, Armenia's president, who was obliged to stand down after two terms.
The prime minister took an unassailable lead with 844,088 votes, giving him 52.7 per cent of the votes counted, according to official results shown on Armenian public television.
Ter-Petrosyan had 344,619, or 21.5 per cent, of votes counted.
Since Sarkisian received over 50 per cent of the vote, he will not have to face a runoff with Ter-Petrosyan.
Voter turnout was 69.25 per cent, the Central Election Commission reported.
Ter-Petrosyan and Sarkisian's other main rival, Artur Baghdasarian, had boosted the prime minister's chances of winning the election by failing to unite ahead of the vote.
Previous elections in Armenia have been followed by days of opposition protests alleging ballot fraud.
Sarkisian has promised to pursue the policies of Kocharian.
His biggest challenges will be a simmering territorial conflict with neighbouring Azerbaijan and frozen ties with Turkey.