Kelly emphasised that there were still improvements to be made in several areas.
"Revelations of extrajudicial killings are evidence that the armed forces' far reaching reforms have not fully taken hold," he said.
Kelly also said that the US was "troubled" by allegations of illegal wiretapping and surveillance by Colombia's Department of Administrative Security.
Colombia's government says that it has worked to eradicate human rights violations, but claims of abuses remain common across the country, where the government has fought rebel groups with ties to the illegal drug trade for decades.
Colombia is the deadliest country in the world for labour rights activists, with 49 killed in the South American nation last year, the International Trade Union Confederation says.
A UN human rights investigator reported in June that soldiers had killed hundreds of innocent civilians, identifying them as opposition fighters from rebel groups to inflate body counts.
Maria McFarland, a senior Americas researcher at Human Rights Watch, criticised the decision by the US.
"The pattern in the past has been of ignoring serious human rights abuses and letting the money flow,'' she said of the US government.
"Clearly this time, even though there are serious abuses, they've decided to certify.''
The move by Washington to release millions of dollars of withheld aid under the certification plan comes days after the US reached a deal with Bogota which will give US forces greater access to Colombian military facilities.
That agreement has been criticised by several Latin American nations, including Ecuador and Venezuela.
Hugo Chavez, the Venzuelan president, has said that the deal is part of a plan by Washington to assert control over the region, claims that the US military has denied.