New Zealand has instructed the defence contractor Trijicon, which is based in Wixon in Michigan, to remove the citations from their future orders of the weapon sights.
Dunne said that New Zealand has 260 such sights, first bought in 2004, which will continue to be used once the codes are removed as they are the best quality available.
The US Marine Corps was said by ABC News, which broke the news of the inscriptions, to have a $660 million contract over multiple years with Trijicon for them to make 800,000 units of the product. Trijicon has other contracts to supply the US amy with the sights.
"We all know of the religious tensions around this issue and it's unwise to do anything that could be seen to raise tensions in an unnecessary way"
New Zealand's defence minister
The Marines have said that they will meet with the company to discuss future deals.
"If determined to be true, this is clearly inappropriate and we are looking into possible remedies," Commander Darryn James, a Pentagon spokesman, told the AFP news agency.
The sights are used on weapons used during the training of Afghan and Iraqi soldiers under contracts with the US army and Marine Corps.
The US-based Muslim Public Affairs Council (MPAC) called on Robert Gates, the US defence secretary, to immediately withdraw the equipment from combat.
"Having Biblical references on military equipment violates the basic ideals and values our country was founded upon," Haris Tarin, MPAC Washington director, said in a statement.
"Worse still, it provides propaganda ammo to extremists who claim there is a 'Crusader war against Islam' by the United States."
US military forces have repeatedly stated that their missions in Afghanistan and Iraq are secular and they have a ban on proselytising.
The inscriptions on the sights refer to passages in the New Testament
One relates to the second letter of Paul to the Corinthians and says: "For God, who commanded the light to shine out of darkness, hath shined in our hearts, to give the light of the knowledge of the glory of God in the face of Jesus Christ".
Wayne Mapp, New Zealand's defence minister, said that with soldiers operating in Islamic nations, the references could be misunderstood.
"We all know of the religious tensions around this issue and it's unwise to do anything that could be seen to raise tensions in an unnecessary way," he said.
The British ministry of defence also announced it had placed an order for 400 of the gun sights but said it had not been aware of the significance of the inscriptions.
David Chater, Al Jazeera's correspondent in the Afghan capital Kabul, said: "It is a rallying cry for the Taliban. It gives them a propaganda tool.
"They've always tried to paint the US efforts in Afghanistan as a Christian campaign."
Trijicon said that it has manufactured weapons sights with Biblical inscriptions for three decades and never before received a complaint.
"We don't publicise this," Tom Munson, the director of sales and marketing at Trijicon, said in a recent interview.
"It's not something we make a big deal out of. But when asked, we say, 'Yes, it's there.'"
The references were originally put on the sights by Glyn Bindon, the company founder who died in 2003, and the practice has continued.