ANSAN, South Korea, Aug 22 (Reuters) - South Korean families who lost loved ones in April's ferry disaster are demanding accountability from the government, but some have grown weary of strident activists adopting their cause for political ends.

The overloaded Sewol capsized and sank on a routine voyage that killed about 300 people, most of them children from the same school, causing an outpouring of grief as well as outrage at President Park Geun-hye's conservative government for what was widely seen as a botched rescue operation.

Four months later, the tragedy is so politically charged that Pope Francis had to answer for wearing a yellow ribbon in support of the victims during his visit to Seoul.

Some family members have tired of the political to-and-fro over proposed legislation to create an independent investigative committee with the right to prosecute. The People's Committee for the Sewol Ferry Tragedy, which supports mourning families, consists of more than 800 civic groups, many of them already critics of Park.

"Everyone is getting exhausted. Most of us like me want to see some kind of closure," the father of one victim told Reuters.

He was speaking during a cigarette break on Wednesday outside a meeting of family members in Ansan, the working class city southwest of Seoul where most of those killed in the disaster had been pupils at the Danwon High School. He declined to give his name and was wary of being overheard.

Another father of a victim said some family members did not want left-wing activists helping them, as it compromised their political neutrality.

"Some of us didn't want to mingle with them, but at that time we were office workers who didn't know how to speak up for ourselves," he said. "So I thought we needed their support."

The mother of a victim, who declined to be identified outside the meeting in Ansan, said she had had no choice but to defer to those championing their cause.

"It is somewhat burdensome that those civic groups are helping us and some people (not tied to the disaster) speak ill of us. But as a mother who lost her child, I have no choice but to follow people who are active in our group because they are doing something that I can't do," she said.

At Wednesday's meeting in Ansan, households overwhelmingly backed their initial position calling for an investigative committee, with 132 out of 176 voting to stick with that demand.

"I HAVE A HEADACHE"

At a makeshift camp of victims' supporters in the centre of Seoul, a father has been on hunger strike but many of more than 100 people gathered on a morning this week appeared to be left-wing activists.

Among the posters and banners at the protest site, one says: "Why is the (presidential) Blue House trying to kill Yu-min's father? President Park Geun-hye, come out and take responsibility!"

Activists have rallied around Yu-min's father, Kim Young-oh, 47, who lost his 16-year-old daughter in the disaster and has been fasting, living off water and salt, for more than 30 days.

"I have a headache. I have a headache because of politicians in South Korea," Kim said after the leader of the main opposition party visited him following a compromise on the proposed legislation that fell short of what many of the victims' families have been demanding.

"We want to find why more than 300 people died unfairly. We want to clarify this and hold a person in charge accountable," said Kim, who handed a letter to Pope Francis on Saturday before a massive open-air Mass in Seoul.

South Korea, after decades of armed standoff with the reclusive, nuclear-armed North and entrenched regionalism, has long been politically polarised. Protest groups and demonstrators, and a heavy police presence, are a staple of everyday life in downtown Seoul, and some among the ferry families' victims and their supporters have complained of overzealous treatment by police.

A man who lost his mother in the accident said the relatives of the 43 victims not part of the school trip also felt sidelined.

"We keep appealing that voices of those who are not Danwon High School victims' families should be heard. But what they say is the number of casualties from Danwon High School is high. They have never asked for our opinion," said the man, who declined to be identified.

Miryu, an official with the People's Committee for the Sewol Ferry Tragedy, acknowledged that it was not close with non-school victims' families, but believed that their interests were the same.

"We think we are fulfilling our responsibilities as citizens to face the families' pain," she said.

The pope, meanwhile, was asked during his return flight to Rome about the yellow ribbon he wore in support of the ferry victims.

"I took this ribbon ... out of solidarity with them, and after half a day someone came close to me and said: 'It is better (you) remove it, you should be neutral,'" he said.

"But listen, one cannot be neutral about human pain. I responded in that way. That's how I felt." (Additional reporting by James Pearson, Sohee Kim and Narae Kim; Editing by Tony Munroe and Nick Macfie)