I have also seen this error when using npt>~30000 in projective transform test (the error was in svd function), and that is why I chose npt=10000 instead of 100000 as for the other tests. If the error you obtain is also in svd, you just have to put npt=1000 or so.

I wrote those tests on a fairly recent 64 bit linux box where the only one that occasionnally fails is the polynomial ord = 4.
Now on a 64bit MacBook Pro some other tests occasionnally fail. I will have to dig into this. Is mldivide supposed to behave differently from one platform to another?

About the "tolower" you added : it is unusefull as "ttype" is always lowercased at line 133. We should remove one of theim.

Also made a small changes so that the functions are not case-sensitive about the function name (use tolower around the option name to start the switch block).

I tried to run the tests you wrote for cp2tform, to make sure my changes did not break anything but I get a memory exausted error or requested size too large (my system is 32bit). Would it be possible to write some smaller tests? If not, would it be possible to have different tests depending on the system?