The actress is the first celebrity to resign from the Oxford-based charity.
The Good Will Hunting star said she was "horrified" by allegations that some Oxfam staff had sex with prostitutes in crisis-hit Chad and Haiti.
Several corporate sponsors including Sainsbury's, Visa, M&S and the Co-op bank, have said they contacted the charity to raise concerns about the claims.
In a statement to The Telegraph, Driver said: "I am nothing short of horrified by the allegations against Oxfam International.
"In no uncertain terms do I plan to continue my support of this organisation or its leaders.
The British actress said she had supported Oxfam for 20 years, adding: "I certainly will not let the abhorrent mistakes of a troubling organisation stop me or anyone else from working with good people in this space to support a population of human beings around the world that needs our help."
Driver, 48, travelled to Thailand and Cambodia in her role with Oxfam and performed at a charity concert in London in 2004, joining stars such as Chris Martin, Razorlight and REM.
In a statement, Oxfam said the actress had been a "devoted Oxfam supporter for many years".
"Her decision to step down as an Oxfam Ambassador saddens us deeply, but we also understand and respect her choice," the charity said.
"As an organisation, we are ashamed by the actions of some of our staff and are more committed now than ever to learn from our mistakes and ensure we uphold the highest levels of safeguarding standards in our work around the world."
Oxfam aid workers are alleged to have used prostitutes in Haiti after the devastating earthquake in 2010 and in Chad in 2006.
More than 1,200 people have reportedly cancelled their monthly donations to the charity following the claims, first published by the Times.
It has been reported that the aid worker at the centre of the scandal, Roland van Hauwermeiren, joined Oxfam after he left another British charity over similar allegations of sexual misconduct.
Van Hauwermeiren resigned from his role with medical relief charity Merlin in Liberia in 2004 after an investigation into sex parties with local young women, according to humanitarian news website Irin.
He denied the allegations but agreed to stand down, it added.
Former aid worker Amira Malik Miller told Irin she raised concerns about van Hauwermeiren and several colleagues to Merlin, which merged with Save The Children in 2013.
Oxfam said it could not find records on its recruitment of van Hauwermeiren but its policy at the time was to get two references including a former employer.
An Oxfam spokeswoman said: "This case further illustrates the need for a sector wide approach to track individuals guilty of misconduct and why we announced on Sunday a package of measures to strengthen our vetting and recruitment procedures."
The Charity Commission said it had contacted Save The Children to establish if it had records of Ms Miller's complaints.
A spokeswoman told Sky News: "This clearly raises issues for UK charities about recruitment, vetting and references for charity personnel, particularly in the context of international aid work to ensure that people who pose a significant risk to charities, their work and reputation are not allowed to infiltrate the charity sector."
Save The Children said it had never employed van Hauwermeiren, who left his role with Merlin nine years before the two organisations merged.
"We are unable to assist any further in this matter," a spokeswoman said.
The Charity Commission has launched a statutory inquiry amid concerns that Oxfam may not have "fully and frankly disclosed material details" when it first investigated the allegations of misconduct in 2011.