if the last letter of a spanish word is either "o" or "a" depending on gender, couldn't this question that gives the option "mia" or "mio" be correct either way depending on what gender the person answering it is?

Not the gender of the person, but rather the gender of the thing possessed (which makes sense because it is a pronoun). So, if the object is masculine, the pronoun changes accordingly. This also happens if it is a plural object - pronoun becomes plural: Los lápices no son míos