In the statement read out at the Hague by the judge, the prosecutor noted "that a person may not be placed in provisional detention unless he is willing to indict in a very short time frame".
The prosecutor considered "the evidence available to him currently is not sufficiently credible to request the maintenance and detention of those persons", the pre-trial judge said.
Rula Amin, Lebanon's correspondent in Beirut, said: "It's a very significant decision, and the Lebanese government has already said that it would follow whatever decision the UN tribunal stated."
The generals have been held for three years and eight months without being charged.
The special tribunal for Lebanon, which began work on March 1 and is based in The Hague, Netherlands, said three weeks ago that Lebanon had supplied a list of those detained over Hariri's assassination to the tribunal charged with trying the suspects.
But a Lebanese investigating judge earlier this month lifted arrest warrants against the four generals jailed since 2005 in connection with the killing.
However, the judge also ordered that the four remain in jail pending a decision by the tribunal on their fate.
Hariri was killed along with 22 others in a bomb blast on the Beirut seafront on February 14, 2005, stirring a political crisis and leading to the withdrawal of Syrian troops in Lebanon after a 29-year presence.
A UN investigative commission has said there was evidence that Syrian and Lebanese intelligence services were linked to Hariri's killing.
Damascus has denied any involvement.