An asylum seeker at Manus Island, Papua New Guinea, one of a 40-man group recently transferred to Manus Island from Christmas Island. Photograph: Diac

Blind compassion is killing the asylum seeker debate. While Tony Abbott entangles his new government in megaphone diplomacy with Indonesia, upsetting our biggest neighbour in the process, refugees are struggling to survive closer to home.

Vast swaths of the Australian public remain hostile towards asylum seekers, and the advocacy groups supporting them need to ask if the strategy they employed over the last decade is partly responsible for it. Accusing the bulk of Australians of racism and discrimination, a common catch-cry of the left, has done nothing except harden hearts and allow media and political elites to ramp up cruel policies towards the most vulnerable souls landing on our shores.

As new arrivals visit Rise for legal advice on their refugee status, mass persecution continues in Fernandez's birth country – despite the foreign minister, Julie Bishop, claiming otherwise. The Australian ran a fawning profile of Bishop last week. In it, she stated that on a trip to Sri Lanka this year she neither saw nor heard any evidence of persecution against Tamils, although statements by human rights group around the world proves the fallacy of this allegation. Fernandez was released from mandatory detention in 2004 after spending time on Christmas Island and at the Baxter detention centre in South Australia. He says that he's proud of the fact that Rise is the first asylum seeker organisation to be run by ex-detainees and people of non-white background. Rise is not funded by the federal government, and routinely refuses offers by the department of immigration to apply for grants.

Fernandez is outspoken and unforgiving towards the vast bulk of asylum seeker support services. Just this week, he sent out a press release damning the Uniting Church and human rights lawyer Julian Burnside for pushing for temporary mandatory detention. "Will the Uniting Church or any other organisations or community groups ever have the courage," he wrote, "to call for humanitarian reception centres and hostels used in the past in Australia that assist in the rehabilitation of a group of people who are fleeing from situations of extreme trauma and cruelty, rather than advocating for the detention and security industry and stop compromising the lives and wellbeing of the refugee community for the sake of PR?"

The group has released a list of the companies making a fortune from privatised detention, and also didn't spare community groups receiving federal government money, such as Anglicare, Wesley Mission and the Salvation Army.

"All human rights groups in Australia use unhealthy emotional attachment to refugees," Fernandez tells me. "I'm against the stereotyping of refugees with sad and crying faces. White guilt is rampant, which often causes people to act to make themselves feel better rather than empowering asylum seekers. I want refugees to have a voice in this debate. I hate how refugees are often told at refugee rallies, by asylum seeker groups, "thanks for coming", which is patronising and crazy when it's white people saying that to brown and black people. Australia is one of the most racist nations in the world. Australia has never accepted that it has a racist past."

Rise hosts a food bank, hip-hop concerts by asylum seekers, English classes, legal advice, publishes art and writing by refugee survivors, has a library featuring books by primarily non-white authors and a drop-in centre for children and adults. There are now 1,060 registered refugees with the group, and the need for such services has never been higher.

One of Rise's volunteers is Nazeem Hussain, a Muslim comedian with a Sri Lankan background who performs with the group Fear of a Brown Planet and is the star of the SBS skit-show Legally Brown. Hussain tells me, "I grew up in a community of refugees and Rise is the only group I know that has the understanding of what they're going through. Too many people are involved in the asylum seeker movement to feel good about themselves instead of helping refugees. Australia has a huge problem with racism, affirmed by both major sides of politics. The intervention against Indigenous people in the Northern Territory is mostly ignored because the racism is accepted."

Over in west Melbourne, I visit the Asylum Seeker Resource Centre (ASRC). Community team leader Jana Favero shows me around the three story office building. She fears the coming three to six months under the Abbott government will tighten the noose around refugee rights and reduces their ability to source legal and community care. ASRC's funding is mostly from private donations and philanthropy, with only 5% from the Victorian government, which ensures they're not at Canberra's mercy.

Favero worries that the "Abbott government could put pressure on the immigration department to discontinue our mostly good relationship between us and them, helping individual refugee cases." ASRC never has enough money for the daily assistance in food, legal advice, housing, english and children's classes and job hunting (for refugees allowed to work). Thousands of asylum seekers remain in limbo, on visas that don't allow them full rights and unsure when that status will change, if ever.

Every day, volunteers cook up a lunch meal for up to 150 asylum seekers with donated food. Refugees have access to a large storage room for food, soap and healthcare products thanks to a needs-based points system. Refugee families are also given assistance in cooking healthy meals. During my visit, I meet Hazara men and women, African asylum seekers and, since it was during school holidays, a few rampaging kids. ASRC becomes an unofficial child care centre at various times during the day, and its office now opens into the night a few times a week to cope with the huge demand for services.

The ASRC's Pamela Curr worries that the new federal government will only increase the secrecy around asylum seekers, a policy ably challenged by the Australian's Mark Day this week. "I know of government-backed contractors signing confidentiality agreements and NGOs forcing staff with poor english to sign agreements with no legal standing", Curr says. "They claim that speaking out in the media is not the way forward, private engagement with the immigration department is better, but it's not born out by the facts." One year after the re-introduction of off-shore processing, she despairs as she still hears stories of young men being raped and abused on Manus Island.

Resistance to the Coalition's policies, coupled with outlining viable alternatives, is surely the best way to reform Australia's dysfunctional relationship with asylum seekers.