Nikolic's Radical Party has aroused the hostility of the EU, Nato and Serbia's neighbours ever since it advocated military intervention to stop Kosovo, a mainly Muslim region, from breaking away from the Serbian state.
A Western-backed plan, which envisages Kosovo's independence, will be debated by the UN Security Council this month.
Nikovic was elected speaker after the Radicals became the largest party in parliament in general elections in January but failed to win an outright majority.
His resignation opens the way for a parliamentary vote on the new coalition cabinet, which consists of pro-Western Democrats led by Boris Tadic, the president, and the Democratic Party of Serbia, headed by Vojislav Kostunica, who is regarded as moderately nationalist.
In his resignation speech, Nikolic warned the new cabinet that if it "peacefully accepts" independence for Kosovo, his Radical party "won't sit calmly and wait".
Serbia's political parties face a Tuesday deadline to form a coalition or hold new elections.