Another 20 prisoners were injured as well as three members of the security forces at Monday's fire at Al-Hair jail on the southern edge of the Saudi capital.
A committee was set up immediately to investigate the cause of the fire, but a high-level security source quoted by al-Hayat newspaper on Tuesday ruled out "any act of sabotage".
The source said: "This sort of accident happens anywhere and a short circuit cannot be ruled out, particularly since the fire started in the afternoon at a time when the electricity supply is overloaded."
The blaze was put out on Monday evening, as about 100 relatives of prisoners gathered anxiously outside, awaiting permission to go in.
Al-Hayat said the facility houses foreign as well as Saudi common law prisoners.
Prison service chief, Major General Ali Hussain al-Harathi said a sponge mattress caught fire in a cell housing 20 inmates, setting off the blaze.
The stricken wing accommodates about 200 inmates.
Three helicopters backed by dozens of ambulances evacuated the casualties.
A massive security cordon was thrown around the prison, and al-Riyadh newspaper reported it casued a traffic jam so bad that there were numerous crashes.
Al-Hair is Saudi Arabia's biggest
Rescue efforts delayed
In London, a Saudi opposition group said security forces stopped and searched civil defence cars and ambulances, delaying rescue efforts and causing the toll to rise.
The security cordon also prevented inmates from fleeing, leaving many to suffocate to death, charged Saad al-Faqih of the Movement for Islamic Reform.
Al-Faqih did not rule out an accident, but said several sources reported the fire was started by inmates "who wanted to draw attention to their grievances, having failed to press their case by other means".
In either case, the fire "was badly managed, with the Interior Ministry taking over rather than civil defence services and giving priority to security over rescue operations," Fagih said.
The Movement for Islamic Reform, which was the first to break news of the fire, said 144 inmates and 40 security men perished in the inferno.