A third contender for the job, independent MP Kamer Genc, withdrew just before the vote started.
Toptan's nomination was largely seen as a sign that the AKP was seeking compromise with secularist forces after a row over its choice of a former member of the Islamic party for the next president.
That dispute, in April, triggered a crisis and led to snap legislative elections on July 22.
"If we listen to each other and achieve reconciliation, there is no obstacle that we cannot overcome," Toptan told the general assembly after his election.
Parliament "will continue to embrace the basic tenets of the republic and ... work towards improving our democracy."
The 64-year-old Toptan served as state, education and culture ministers under centre-right conservative parties before joining the AKP.
He served as the head of the parliamentary justice commission before last month's elections.
The AKP won a landslide victory to secure 341 of the 550 seats in parliament and a popular mandate to govern Turkey for a second five-year term.
In terms of protocol, Toptan will now be the country's number two after the president.
Toptan's smooth election comes just before potentially tense elections in parliament to choose the country's next president.
The AKP was forced to bring legislative elections forward from November after it twice failed in April and May to elect Abdullah Gul, the foreign minister, as head of state when the secular opposition boycotted the vote in parliament.
The crisis worsened with a threatening statement from the army and mass street protests against the prospect of a president from the AKP, which secularists accuse of seeking to erode the separation of state and religion.
Gul has indicated that he could run for the presidency again.
But AKP sources and newspaper reports said on Wednesday that the party was divided over whether or not to re-nominate him on fears that such a move would revive tensions with secularist forces and the military.