"People are still dying, the rebels and government are still clashing," he said. "We have given our support, but there are no results on the ground."
Sudan has demanded tight restrictions on any UN movements in Darfur, stalling UN plans to bolster the AU force.
The UN Security Council is due to discuss President Omar Hassan al-Bashir's objections on Thursday.
Rwanda felt "strained" in the current situation, Kagame said, and unless more resources were committed, he added, it was "considering its relevancy" to the mission.
"We are in consultation with our partners regarding our stay there, but once we take a decision of withdrawing our troops it will be done once and for all," Kagame said.
Rwanda says it has spent $12 million on its deployment so far, but has only received half of that back from the AU.
"If we are not getting the best support and our presence is not benefiting those in the conflict, it is not benefiting us either to be there," Kagame said.
The AU is facing a shortage of money and equipment for its violent Darfur assignment.
At least 200,000 people have died in Darfur, 4 million need emergency aid and 2.5 million are in makeshift camps.