The indices returned are always in [0, height) x [0, width) before flattening,
even if padding is involved and the mathematically correct answer is outside
(either negative or too large). This is a bug, but fixing it is difficult to do
in a safe backwards compatible way, especially due to flattening.