About 50 fighters entered the remote, mountainous Tsuntinksy region near Dagestan's border with Georgia around midnight, interior ministry officials told the RIA Novosti news agency on Monday.
A state of emergency has been declared in the region, troops have blocked it off and additional forces have been sent for, news agencies reported.
"The rebels are right now blocked near the village of Shauri," an official with Dagestan's interior ministry said, adding that elite forces from the ministry were heading there.
The official said the attackers had seized 30-40 hostages at a school, but Basyr Magomedov, head of the Tsuntinsky district administration, said on state-run Rossiya television that the attackers were holding an unspecified number of hostages in a 50-bed hospital.
Another Dagestani emergency official said the attackers had left Shauri, taking three hostages with them.
"Our people are trying to make contact with them (to find out) what they want," Magomedov said.
There were conflicting reports as to whether the rebels had crossed into Dagestan from Chechnya on their way to Georgia, or whether they entered from Georgia and were heading towards Chechnya.
The battle ensued after the rebels probably lost their way in the snowy mountains and stumbled on local residents who alerted the authorities, military officials said.
In August 1999, a group of Chechen rebels crossed into Dagestan and took over two villages in a region north of Tsuntinsky.
The incursion lasted several weeks and sparked the second Russo-Chechen war in 1999.
The then prime minister Vladimir Putin sent Russian troops into Chechnya on 1 October 1999, for what was supposed to be a lightning-strike operation to root out "terrorists".