I don't really know about using the reference as a hash key, someone else will have to answer this question. There is however Scalar::Util refaddr function which you could use to explicitely use the reference's address as a key to a hash (or even as an array index).
Edit: Better use hashes for this, performance with an array might be much worse since the memory addresses can get very huge.