WASHINGTON (CNN) -- A major United Nations anti-racism conference was thrown into further disarray Sunday when more countries joined a U.S. boycott amid concerns it was developing into a platform for attacking Israel.
The Congressional Black Caucus criticized the boycott, saying President Obama's decision "set the cause back."
Australia and the Netherlands were the latest to pull out of next week's meeting in Geneva, Switzerland, as a dispute gathered pace over a document said to single out Israel for racism.
Canada, Israel, Italy and Sweden have also announced they are boycotting the conference aimed at creating a global blueprint for tackling discrimination. Britain says it will attend.
The U.N. High Commissioner for Human Rights, Navi Pillay, said she was "shocked and deeply disappointed" at the boycott.
"A handful of states have permitted one or two issues to dominate their approach to this issue, allowing them to outweigh the concerns of numerous groups of people that suffer racism and similar forms of intolerance to a pernicious and life-damaging degree on a daily basis all across the world, in both developed and developing countries," she said in a statement. "These are truly global issues, and it is essential that they are discussed at a global level, however sensitive and difficult they may be."
Washington says despite numerous redrafts of the controversial document, due to be adopted at the conference, it remains unhappy, and says it violates American principles of free speech.
The document will reaffirm anti-discrimination commitments agreed at a 2001 meeting in Durban, South Africa. The United States objected to the 2001 agreement -- joining Israel in walking out of the Durban meeting.
The U.S. State Department said the current document "prejudges key issues that can only be resolved in negotiations between the Israelis and Palestinians."
Australia's Foreign Minister Stephen Smith echoed the concerns on Sunday, saying that Israel was being unfairly targeted.
"Regrettably, we cannot be confident that the Review Conference will not again be used as a platform to air offensive views, including anti-Semitic views," he said.
"Israel regrets that the conference ... has once again become hostage to one-sided, non-constructive politicization and biased rhetoric," Israel's ambassador to the United States, Sallai Meridor, said in a statement Sunday.
President Obama said Sunday the 2001 conference became "a session through which folks expressed antagonism towards Israel in ways that were often times completely hypocritical and counterproductive."
He said the United States had warned it would not attend this year's conference if the document was not sufficiently altered in advance.
Obama said the United States is hoping to partner with other countries "to actually reduce discrimination around the globe, but this [conference] wasn't an opportunity to do it."
The boycott has caused concern among anti-racism campaigners in the United States.
The Congressional Black Caucus said it was "deeply dismayed" by the decision made by the nation's first African-American president, saying it was inconsistent with administration policies.
"Had the United States sent a high-level delegation reflecting the richness and diversity of our country, it would have sent a powerful message to the world that we're ready to lead by example," a statement from the group said.
"Instead, the administration opted to boycott the conference, a decision that does not advance the cause of combating racism and intolerance, but rather sets the cause back."
Meanwhile, Iranian President Mahmoud Ahmedinejad, whose past comments on the Holocaust and Israel are likely to overshadow his contributions to the anti-racism conference, has reportedly confirmed his attendance.
CNN's Scott Spoerry contributed to this report.
All About Racism and Bigotry • U.S. Department of State • Middle East Conflict
From the Blogs: 
Sit tight, we're getting to the good stuff
