"To be honest, you could walk away from all of this tomorrow," Brown said in the Guardian newspaper's Saturday edition.
"I'm not interested in what accompanies being in power. It wouldn't worry me if I never returned to any of those places
- Downing Street ... That would not worry me at all. And it would probably be good for my children."
The expense scandal, which tarred all three of Britain's main political parties, prompted several lawmakers to resign and sparked a failed challenge to Brown's leadership. It was also thought to have brought the Labour Party its lowest share of votes since World War I in local and European elections earlier this month.
Despite flagging popularity in the polls, Brown has insisted he has "a job to do." Brown waited a decade for his predecessor, former Prime Minister Tony Blair, to step aside. Blair was known as charismatic political orator and operator.
In the rare personal interview, Brown said he lacked skills in both categories.
"Look, find weaknesses in me, criticize me for my weaknesses - I'm not as great a presenter of information or communicator as I would like to be ...," Brown said.
He also said it had been a revelation that politics was less about ideals and "more about maneuvers."
"I'm not sure I'm that good at it ... I don't actually think I am very good at it at all."
Brown has been challenged during his short tenure by what he called "two earthquakes
- one economic, unparalleled since the war, one political, the biggest parliamentary scandal for two centuries."
He has suffered two personal tragedies in his life - the death of his infant daughter in 2002, and the loss of the sight in one eye when he was a teenager
- but he said his recent political challenges have been hard felt.
"I've been through lots of different problems over the last 20 to 30 years, but this is one that's been more in the public eye. But you stop thinking about who you are and think about what you've got to do."
Britain is scheduled to have a general election by mid-2010.