The warnings come as leaked documents revealed staff contracted by the Australian government have deliberately tried to make conditions at the facility worse and drive people out.
Refugees and asylum seekers on Manus Island have been told they must leave the Australian-funded detention centre so it can close.
A Papua New Guinea official reportedly told detainees demolition will begin as early as May 28 - more than a year after the PNG Supreme Court ruled the detention centre was "illegal and unconstitutional".
SBS has obtained a recording of the unnamed official informing the more than 800 detainees who remain they must decide to either return to their country of origin, settle in the PNG community, or relocate to a so-called "transit centre" near the main town of Lorengau.
Leaked documents published in The Guardian reveal the Australian contractors that operate the centre planned to worsen conditions for those who remain at the centre.
The documents state:
"Conditions for refugees at East Lorengau refugee transit centre should be more attractive than for refugees at Lombrum Regional Processing Centre."
Suggestions include the division of those with refugee status from those without.
The latter would be moved to a compound at the western end of the detention centre, "because it has no air conditioning", in order to "provide a visible change of circumstances to refugees."
There were also suggestions to “remove potentially negative influences of persons who are on a departure pathway".
Watch: Human Right Law Centre on Manus Island
Iranian asylum seeker Benham Satah is one of those who remains at the centre.
He told SBS World News he and other detainees were facing an impossible choice.
"We are going through one of the worst dilemmas ever in our lives,” he said.
“We really don't know what to do. The mood is not very stable for a couple of days, since PNG officials announced the movement and relocation (to Lorengau), the mood is not very well inside the compound.”
Mr Satah, who witnessed the murder of his roomate and friend, Reza Berati in 2014, says many are concerned for their safety in the PNG community, fearing attacks from locals.
"Policemen are supposed to protect people,” he said.
“Don't they really know what happened on Manus? Don't they know about the crime that happened here, how many people died on Manus Island?"
Some are facing pressure to abandon their refugee claims and return to the countries they fled from, before the deadline for an Australian government incentive of up to $20,000 expires at the end of August.
In a series of text messages, Iranian refugee Behrouz Boochani has told SBS News he believed there would be violence if people were forced to relocate.
"The Australian government is trying to kill the refugees or put them in a situation [where they have] to go back to their countries,” he said.
“I'm sure if they move the refugees by force into the PNG community, more people will die.
“The refugees plan to resist because of their lives and their future. Of course there will be violence. These people have nothing to lose."
Papua New Guinea's police commander confirmed earlier this week that soldiers fired shots directly into the Manus Island detention centre last month.
Listen: Benham Satah speaks to SBS World News
Greens Senator for Tasmania Nick McKim told Sky News those who remained at the centre did not have any viable options.
"They've deliberately made conditions harsh at the Lombrum detention centre in order to try to force people to return home and face whatever conditions they were fleeing back at home,” he said.
“And that's the definition of torture under international law."
The Human Rights Law Centre's Daniel Webb says the Australian government needs to provide viable options for the men who remain on Manus Island.
"The government has known for some time that Manus is a dead-end. But instead of finding a humane way forward, the government's been deliberately worsening conditions to try to pressure these guys to give up and go home, or to risk their safety outside the camp in the PNG community.
United States officials have been on Manus island in recent weeks, assessing which and how many refugees the US will accept for resettlement under a controversial deal to take an unspecified number of refugees from Manus and Nauru.
But there is no guarantee how many - if any - refugees the US will be accepted from Manus Island under the deal.
