At least 10 members of the research team were also kidnapped in the attack on Wednesday.
One local official said all ten kidnapped people have been rescued, and local media reported that five oil workers were taken to the Borno State Specialist Hospital.
The military personel were protecting the convoy, which was conducting research near Maiduguri, the capital of Borno State in northeast Nigeria.
A humanitarian worker in the area said there were about 30 people on the exploration team, which was partially made up of staff members from the University of Maiduguri.
Al Jazeera's Ahmed Idris, reporting from Abuja, said an operation is under way by the Nigerian military to rescue those who survived the attack.
The Nigerian National Petroleum Company (NNPC) has been surveying for more than a year for what it says could be vast oil reserves in the Lake Chad Basin.
Nigeria relies on oil for two-thirds of its revenue. The NNPC is trying to reduce its reliance on crude oil from the southern Niger Delta, where attacks by armed groups cut production by more than a third in 2016, deepening the recession in Africa's biggest economy.
In May, the NNPC said it would resume oil exploration in the northeast "on the heels of [the] improved security situation".
At least 62 people have been killed in Maiduguri and the surrounding area since early June. Seventeen people were killed in the city in one week this month.
Last week, Nigeria's top army commander gave his soldiers 40 days to capture Abubakar Shekau, the leader of Boko Haram.
Boko Haram's eight-year armed campaign in the region has killed more than 20,000 people and forced up to 2.7 million to flee their homes.