The Parliamentary Joint Committee on Human Rights has published 135 submissions from individuals and organisations for its inquiry into Freedom of Speech in Australia.
Many opposed any changes to 18C of the Racial Discrimination Act.
The Ethnic Communities Council of Victoria made a joint submission with 83 other religious and community groups, including Muslim, Christian, African, Sikh and Jewish organisations.
Council Chair Eddie Micallef said many migrants, especially the most recent arrivals, would be left feeling very vulnerable by the watering down of section 18C.
"I think it just gives those who have a racist bent some encouragement," Mr Micallef told SBS.
"It will encourage that group and it will potentially cut across the social cohesion that we've been able to build up in Australia over many years," he said.
The inquiry will also investigate whether the complaints-handling procedures of the Australian Human Rights Commission should be reformed.
A submission from the Executive Council of Australian Jewry recommended there could be some changes to that process.
But the Council's chair, Peter Wertheim, said the substance of 18C should remain unchanged.
"We believe that section 18C, when read together with the exemptions in Section 18D, provides the right balance between freedom from racial hatred on the one hand and freedom of expression on the other," Mr Wertheim said.
But others argued Section 18C is obsolete.
"I think it just gives those who have a racist bent some encouragement."
In its submission, the "anti-Islamisation" minor political party, Australian Liberty Alliance, argued the current legislation constitutes a violation of common law freedoms.
It also said the laws contravene Article 19 of the Universal Declaration of Human Rights, which states that everyone has a right to freedom of opinion and expression.
The Alliance's Ralf Schumann told SBS, other legal protections against discrimination already exist.
The Media Entertainment and Arts Alliance argued in its submission the current terminology had become increasingly confusing.
It recommended a review of the legislation should consider replacing "insult" and "offend" with "vilify."
The parliamentary inquiry was ordered by Prime Minister Malcolm Turnbull.
A similar move was dropped by former Prime Minister Tony Abbott two years ago in the face of strong community opposition.
Labor has accusing Mr Turnbull of being swayed by right wing elements in his party.
Labor Senator Doug Cameron told reporters in Sydney, community groups have been telling the ALP they're angry about the push to weaken the legislation.
"We haven't been mobilising the ethnic communities, the ethnic communities have been coming to our offices again all over the country complaining about 18C going," Senator Cameron said.
"Malcolm Turnbull (is) being led by the nose by the extreme right wing of his party," he said.
"He's simply not the man that people thought they had voted for."
Some migrant lobby groups are warning the Coalition could lose community support over the issue.
Kenrick Cheah, President of the Chinese Australian Forum, said there could be a backlash in some Coalition seats.
"It's a close parliament and with certain seats, especially NSW, certain Coalition seats especially, Reid, Bennelong, Banks, there's a big multicultural population and they don't like racism," he said.
Mr Cheah also works for NSW Labor, but said the Forum has members of both major political parties.
Some Liberal MPs, including Liberal MP for Bennelong John Alexander have questioned the need for change.
But Liberal Senator Scott Ryan denied there was a split over the issue.
"I would not use those words at all," Senator Ryan said.
"The Coalition has always had people of various views and we are quite happy to debate those views," he said.
