Kurram's top administrator, Arshad Majeed Mohmand, confirmed Monday's strike and said three missiles were fired.
Abdul Rahim, a cleric in Sarpul, a mountainous area on the outskirts of Baggan village in Kurram, said he saw around 15 bodies pulled out from the rubble and 20 wounded.
The building that was hit was formerly used by Afghan refugees' children, but fighters moved in around two years ago, according to villagers.
Monday's attack was the fourth linked to the US since Barack Obama took office as US president last month.
Pakistan's civilian government, elected a year ago, and the army have complained that missile strikes are counterproductive and have fanned an insurgency across northwest Pakistan.
Shah Mehmood Qureshi, Pakistan's foreign minister, said this week that the attacks were "happening without any understanding and it is affecting our sovereignty, and we think that it is causing collateral damage".