MPs from both the government and the opposition have been criticised over their expenses, which have ranged from claims for moat cleaning to the rental of pornographic films.
In her resignation letter, Ussher said her actions "have been in line with HM Revenue and Customs guidance and based on the advice of a reputable firm of accountants".
"I did not do anything wrong ... Neither have I abused the allowance system of the House of Commons in any way"
former British treasury minister
"It is with the greatest regret I have decided to resign from the government," she said.
"After careful consideration I arrived at this decision because I do not want to cause you or the government any embarrassment.
"I did not do anything wrong ... Neither have I abused the allowance system of the House of Commons in any way."
Ussher had previously said she would stand down from parliament at the next general election, citing family reasons.
Ussher was only appointed to her role as exchequer secretary to the treasury earlier this month after Brown reshuffled his ministers following results in European elections and challenges to his leadership.
Previously she had served as a junior pensions minister, and before that in another finance ministry role.
Sarah McCarthy-Fry, MP for Portsmouth North, will replace Ussher at the treasury.
"The prime minister has accepted the resignation of Kitty Ussher. Sarah McCarthy-Fry has been appointed as her replacement. A successor to McCarthy-Fry will be announced in due course," Brown's office said in a statement.