The OED cites several instances of "no skin off my nose" and various other places, such as "bugle" for nose, "off me" and "off my ass." It did not mention "off my back." It means, of course, "It's all the same to me," or "I don't care." SS