I'm trying to write my own RSA implementation using the textbook approach — which I know is not optimal — of picking primes $p$ and $q$, then computing Euler's totient $phi$, then randomly picking an 'e' which is relatively prime to $phi$, and then computing a $d$ such that ($d*e$) mod $phi$ = 1.

2 Answers
2

In modular arithmetic all of $a + k \cdot m$ with modulus $m$ and integral $k$ are equivalent. You need to solve this equation modulo $\phi$. So $-x$ is equivalent to $-x + k\cdot \phi$ with a sufficiently large $k$ to make it positive.

In addition to what CodesInChaos stated (which does apply): the code you linked to does not support integers bigger than the maximum for the C int type, typically $2^{31}-1$. This is not enough in an RSA context, and may cause problems similar to what you have. You can

switch to a language with built-in support for big integers, like Python, Java..