The Daily Mail newspaper quoted a security source as saying: "There was no British involvement and they didn't know the name of the target. But they were told these people were travelling on UK passports."
UK officials have previously said they knew nothing about the operation until the Dubai authorities released details of the assassination.
Forged British, Irish, German and French documents were used by 11 people named by the United Arab Emirates as suspects.
At least six of the names released by Dubai authorities match those of Israeli-British citizens, all of whom deny any connection to the killing.
Speaking to Al Jazeera, Robert Fisk, the British journalist, said that the use of the passports should be a cause for concern, but that the UK was unlikely to pursue the issue too far.
"At the end of the day, relations with Israel matter more than the murder of a Hamas man, even though the rights of British citizens have been infringed and endangered by the use of their passports," he said.
"We want to know how these passports left the hands of the UK authorities, not just how they got into the hands of what we presume is Israel.
"This is a very serious matter because the lives of every European are now being impinged upon by this case ... How do European countries guard their own citizens when this happens?"
The UK and Irish governments on Thursday summoned the respective Israeli ambassadors to explain the use of the passports.
David Miliband, the British foreign minister, described the use of six British passports as an "outrage".
Paris also demanded an explanation of how a French passport came to be among those used by the suspects.
"We are asking for explanations from Israel's embassy in France over the circumstances of the use of a fake French passport in the assassination of a Hamas member in Dubai," the French foreign ministry said in a news briefing.
Israel has insisted it maintains a "policy of ambiguity" on intelligence matters.
Lieutenant General Dahi Khalfan, the Dubai police chief, who said he was "99 per cent" certain that Mossad were behind the assassination, said Meir Dagan, who heads Mossad, should be arrested if the agency is shown to be behind the killing.
"If the Mossad were proven to be behind the crime, which is most likely now, Interpol should issue a Red Notice for the head of the Mossad because he would be a killer," he said in an interview with Dubai TV.
Police in Dubai have already released closed-circuit television footage of 11 individuals believed to have been involved in the killing of al-Mabhouh at the Al-Bustan hotel.
Eighteen people are now thought to have been involved in the assassination.
But Rami Igrah, a former Mossad unit leader, told Al Jazeera the assassination appeared to be a "frame-up" and that it "looks like somebody is trying to delegitimise Israel again".
"I have to cast doubts on everybody's tendencies to attribute this operation to Mossad," he told Al Jazeera.
"There are several elements in this story. The first thing, these guys didn't miss an opportunity for a photo op in Dubai. The second, more disturbing, seven Israeli citizens found themselves running to Israeli television stations and saying these are our names on these passports.
"Thirdly, there are two Palestinians arrested in Jordan and deported to Dubai and at least one Palestinian belonging to Hamas arrested in Damascus in conjunction with this operation. And all of us know it is very easy to go to the internet and steal anybody's identity if we are a professional organisation.
"On these standings alone, I would cast doubt on everybody's conviction [that] this is the Mossad. It doesn't look like the Mossad."