Luis Moreno-Ocampo, the prosecutor, said he was not "targeting any particular suspect" but would be guided by "the evidence that emerges".
The prosecutor said it was the first time he had opened an investigation in which allegations of sexual crimes far outnumber alleged killings.
"The information we have now suggests that the rape of civilians was committed in numbers that cannot be ignored under international law," he said in a statement.
The prosecutor's office has received hundreds of testimonies from rape victims, including elderly women, young girls and men.
At least 600 rape victims were identified from the five-month conflict, according to the court. Many were reportedly sexually assaulted in front of their families, and, after being abused, were shunned by their communities.
The case was referred to The Hague, where the world's first permanent war crimes court is based, in December 2004 after the Central African Republic's supreme court said the country lacked the capacity to prosecute such cases.
The court can only step in once a member state is unwilling or unable to carry out the prosecution of war crimes, crimes against humanity or genocide.
The investigation is only looking at crimes committed in 2002 and 2003 but the prosecutor said he is continuing to monitor the current situation in the country.
"There are worrying reports of violence and crimes being committed in the northern areas of the country bordering Chad and Sudan," Moreno-Ocampo said.
"In the interests of deterring future violence and promoting enduring peace in the region, we have a duty to show that massive crimes cannot be committed with impunity."
Tuesday's announcement comes just weeks after a rebel group signed a peace accord with the government and agreed to demobilise hundreds of child soldiers.