Voter worries about the economy and healthcare reform appear to have helped Brown.
In Washington, Robert Gibbs, the White House press secretary, said the president does not believe healthcare reform will fail if Coakley loses.
'Surprised and frustrated'
But Obama is "both surprised and frustrated" and "not pleased" by the closeness of the Massachusetts race, Gibbs said.
Polls close at 8pm (0100 GMT on Wednesday) and results could take a few hours to emerge given the expected close vote.
Reflecting Wall Street's expectations on healthcare reform, investors drove health insurance and drug company shares higher, betting a Brown victory would at least slow Obama's healthcare plans.
Hospital companies, which may gain more insured customers under health reform, saw their shares slump.
"If Brown wins, it is our view that Obamacare will not pass congress," Avik Roy, a healthcare analyst with Monnes Crespi Hardt, said in a research note.
Edward Kennedy, the Democrat who held the Massachusetts seat for almost 47 years, died in August of brain cancer.
Fellow Democrat Paul Kirk was appointed by the state's governor in September to occupy Kennedy's seat and will remain in the senate until a winner is sworn in.
Massachusetts last elected a Republican to the senate in 1972, but the weak economy and doubts about Obama's healthcare overhaul have moved some voters to abandon party loyalties at a crucial juncture for the president and his Democratic party.
Democrats now control 60 votes in the senate to 40 for the Republicans. The loss of one seat could hamper the Democrats' ability to cut off debate and proceed to a vote on the planned healthcare overhaul.
Brown has promised to be the Republican's 41st vote to kill healthcare legislation.
More broadly, an upset in Massachusetts, or even a narrow win for Coakley, would raise the spectre of large losses for Democrats in midterm congressional elections in November.