Most of the vessels come from neighbouring Indonesia, whose boat crews say they are being locked out of traditional fishing grounds by Canberra's tough stance.
Nelson said illegal fishing boats were increasingly sophisticated and some were engaging in dangerous measures, such as using large sharpened poles and throwing "missiles", to avoid arrest.
He said he asked for the stronger rules of engagement after an Australian sailor was left hanging from the stern of an illegal fishing boat as it tried to escape.
The navy, which has several patrol boats, a frigate and air force support to hunt for illegal fishing boats, has captured a record 357 of them since January in waters off the northern and northwest coast.
Many of the illegal fishermen were hunting sharks for their lucrative fins.
Under Australian laws, all captured illegal fishing boats are impounded and later destroyed, while their captains face hefty fines and jail terms.
The government in May announced an extra A$500m ($394m) to boost security around its remote northern border with extra surveillance flights and sea patrols.
Australia and Indonesia have also agreed to joint naval border patrols, to crack down on illegal fishing and people smuggling from Indonesia, as part of a new security treaty signed on November 13.
Nelson said the new rules would also include the use of tear gas and pepper spray.
"It is extremely important that anybody who comes to this country seeking to steal our fish and breach our sovereignty knows they will be met with a very strong, disciplined Australian navy," he said.