The Dignitas centre for assisted dying in Switzerland. Photograph: David Levene for the Guardian

Two and a half months ago, my grandfather died. He was 84. He was not unwell in any specific way. He had no cancer and had only spent short periods of time in hospital. He was just old, worn out and over it.

During his last week or so, he expressed a wish to die many times. He once asked his doctor, with no small amount of distress, how it was possible for a person's body to be dying but not their mind. He had reached a point where dying was what he wanted, and we wanted that for him too.

None of us, including his doctor, could do anything to assist him with his wishes because neither euthanasia nor assisted suicide are legal in Australia. With the voluntary assisted dying billvoted down by Tasmanian parliament this week, it doesn't look like that will change any time soon.

For many years now, I have been opposed to legalising assisted death in any form. My recent experience with my dear pa Jack has not changed my mind.

As a disabled person, I'm accustomed to conversations about quality of life and dignity. Specifically, I'm accustomed to assuring people that my life is worth living. I'm short statured, a wheelchair user, and I frequently have bone fractures. All the visual cues that make me "the other" are front and centre. People make all sorts of assumptions about the quality of my life and my levels of independence. They're almost always wrong.

I've lost count of the number of times I've been told, "I just don't think I could live like you," or "I wouldn't have the courage in your situation," or, my favourite one to overhear (and I've overheard it more than once), "You'd just bloody top yourself, wouldn't you?".

What we as a society think we know about what it means to live as a disabled person comes from cultural representations of disability seen through a non-disabled lens. And we, as people with disability, rarely get to tell our own stories.

Also, social attitudes towards disabled people come from a medical profession that takes a deficit view of disability. This is my major concern with legalising assisted death; that it will give doctors more control over our lives.

As a disabled person who has had a lot to do with the medical profession, I can tell you that this is the space in which I've experienced some of the very worst disability prejudice and discrimination. Doctors might know about our biology, but it doesn't mean they know about our lives.

Media reports on assisted dying feed these misconceptions. ABC News reported this week on the case of Barbara Harling, a Queensland woman with motor neurone disease who said that she would consider moving to Tasmania if the voluntary assisted dying Bill had passed. Harling is quoted as saying: "Well, let's put it this way. I can use my left hand, my right hand is just about useless. If I can't use my left hand to wipe my bottom, then I can do nothing else for myself. That means someone has to do everything for me. I couldn't bear to live like that."

The thing is, a lot of people do live like that. I know many, many people who depend on personal assistants for all of their daily living tasks, some of them requiring 24 hour care. Having to rely on someone else to wipe your bum may not be something anyone aspires to, but I'm quite sure it's never killed anyone. Perhaps our discomfort with this kind of thing is why we don't hear the counter view in reports about assisted dying.

Often we hear supporters of euthanasia and assisted suicide talk about wanting to avoid the pain and suffering that often comes with imminent death. But more often, we hear stories like Barbara Harling's, which are more about wanting to avoid a loss of autonomy and independence. Indeed, in the final few years of my grandfather's life it became very apparent that he was not a man who was afraid of dying. He was, however, deathly afraid of disability.

In a two-part audio documentary for the BBC World Service, disability campaigner Liz Carr travelled through Europe and the United States where various forms of assisted death are legal. In the US State of Oregon, where voluntary euthanasia has been practiced legally for over a decade, a recent report on the death with dignity act confirmed that the majority of people using the act do so not because of concerns about pain, or pain control, but instead about losing autonomy and dignity. Carr reasons, "these concerns have as much to do with disability as they do terminal illness, which perhaps is why disabled people like me see assisted death as relevant to them."

Marilyn Golden from US campaign group Not Dead Yet thinks we are asking the wrong questions when it comes to assisted death: "We have to ask, do people with disabilities have true choice and self determination, in terms of living outside of nursing homes? In terms of housing that is truly affordable and accessible? In terms of the kind of services that really allow them to lead meaningful lives? In many cases, no."

Perhaps what we should be asking is, if these services and supports were guaranteed, and we could be assured that we wouldn't become dependent on our family members for assistance, would people still see disability as a fate worse than death?

As an opponent of legalising assisted death, watching my grandfather wanting to die and being unable to give him the release he desperately wanted was heartbreaking. And yet I still believe we need more robust conversation if we are to pass a law that truly balances the wishes of those who want to die with adequate safeguards for those who don't. Before we can talk about death with dignity, we need to ensure that all people, regardless of age or disability, can live with dignity. We're not there yet.