Elllison, a Democrat congressman for Minnesota, shrugged off controversy over his use of the Quran instead of the Bible.
"It was good, we did it, it's over, and now it's time to get down to business," he said.
Asked if he was relieved to have the ceremony behind him, Ellison said, "Yeah, because maybe we don't have to talk about it so much anymore.
"Not that I'm complaining, but the pressing issues the country is facing are just a little bit more on my mind right now."
Ellison was born a Christian but converted to Islam in college.
Republicans have questioned Ellison will succeed in juggling his loyalty to the Ummah, or the global Muslim community, with his loyalty to the US and its constitution.
First woman speaker sworn in
Nancy Pelosi, who helped engineer the Democrat's midterms victory, was also sworn in on Thursday as the first woman to lead the House of Representatives.
The leader of the minority Democrats the past four years, she now is the highest ranking woman in the US government, second behind only the vice president in the line of succession to George Bush, the Republican president.
"This is an historic moment for the Congress, and for the women of this country," Pelosi declared after taking the gavel.
"It is a moment for which we have waited more than 200 years .... For our daughters and granddaughters, now the sky is the limit."
"By electing me speaker, you have brought us closer to the ideal of equality that is America's heritage and hope," Pelosi told colleagues.
Democrats call for Iraq withdrawal
The Democrats have said they will use their control of congress to push for a phased withdrawal of US troops from Iraq.
"No issue in our country is more important than finding an end to this intractable war," Harry Reid, the new senate majority leader.
"Completing the mission in Iraq is the president's job, and we will do everything in our power to ensure he fulfills it."
Bush has said he will announce a revised strategy next week for the war, which has claimed the lives of more than 3,000 members of the US armed forces.