Brunei's Sultan Hassanal Bolkiah and Malaysian Prime Minister Mahathir Muhammad met on Friday to try resolve who should control the lucrative area.
But after the meeting Malaysian Foreign Minister Syed Hamid Albar said an offer Kuala Lumpur made in May was the best proposal it could make.
Sources say the May offer involved an oil production-sharing agreement.
Syed Hamid added it is now up to Brunei to consider the offer.
"The prime minister has made it very, very clear that we have submitted our proposal and that proposal is the maximum we can live with," Syed Hamid said.
But Brunei's UK-based advisers have warned a production-sharing pact may mean conceding Malaysia has territorial rights.
"The prime minister has made it very, very clear that we have submitted our proposal and that proposal is the maximum we can live with"
Syed Hamid Albar
Malaysian Foreign Minister
Brunei, a tiny nation of 330,000 people surrounded on all sides by Malaysia, lacks the muscle to back up its claim.
The stakes in the dispute shot up last year after a sizeable discovery was made in a Malaysian controlled-area near the disputed waters.
Preliminary estimates put reserves at 400 million to 700 million barrels.
Exploration activities in the disputed area were halted earlier this year after the Malaysian navy chased off a drilling team belonging to a consortium led by France's Total.
Brunei's exclusive economic zone stretches 200 km offshore, but Malaysia bases its claim on possession of some tiny islands and reefs in the South China Sea.