Roland Koch, current Hesse premier and a key Merkel ally, won 36.8 per cent of the vote, while Andrea Ypsilianti of the SPD took 36.7 per cent.
Support for Koch fell by 12 points from the last state election in Hesse in 2003, in a strong test to the CDU before federal elections in September 2009.
Political haggling over the leadership of Hesse is likely to affect further the fractious federal coalition government.
In another state vote in Lower Saxony on Sunday, the CDU's candidate had a comparatively easy victory over the SPD, but still lost support compared to the previous election in that state.
The result in Hesse means that the CDU and SPD have 42 seats each in that state.
"I think it is clear - the [Christian Democratic] Union must recognise this too - that people want more social justice," Kurt Beck, the SPD's national leader, said.
"We would have liked a better result in Hesse," Ronald Pofalla, CDU general secretary, said in Berlin.
Koch acknowledged that "the result is not easy for us".
|The loss of support in Hesse and Lower Saxony|
is a warning to Chancellor Merkel [AFP]
Leading newspapers described the result as a disaster for Koch, whose campaign on crime and immigration was backed by Merkel.
"For Angela Merkel, the Hesse election must be a shock which will have a long-term effect," wrote top-selling Bild newspaper.
Koch was criticised by immigrant groups for saying that Germany should impose tougher punishments for youth crime – while pointing to an attack on a elderly German man by two immigrants.
Finely balanced results for three smaller parties mean it is unclear whether the CDU or SPD will be able to lead a coalition in Hesse.
Senior figures from both regional and national parties are set to meet on Monday to negotiate what to do next.
With such a finely balanced vote result, neither the CDU or the SPD is willing to concede defeat.
Furthermore, the Left Party, a far-left grouping of ex-communists and disaffected SPD members, won enough votes to enter parliament in Hesse, which could further complicate efforts to form a coalition in the state.
"Building a coalition in Hesse is going to be incredibly difficult with these results," Richard Hilmar, a pollster, said.
The CDU and SPD parties disagree on most areas of policy in the federal 'grand coalition', which has been in place since Merkel took power in 2005.
"We fear any zeal for reform which remains in Berlin's grand coalition will now be very much weakened," Anton Boerner, head of the BGA exporters' association told Handelsblatt newspaper.
In the election in the northwestern state of Lower Saxony, Christian Wulff, CDU premier, retained power in a coalition with the liberal Free Democrats (FDP).
However, he also saw his support drop. The CDU won 42.5 per cent of the vote - down from 48.3 per cent in 2003.