Just a guess as I don't have a Duo2 to test, but do you need to set SDI1 as an output or should the duplex mode do that for you? Both internal and external keying work as expected with my Extreme3D card but that does not reverse any connections, of course. Also, have you tried feeding the card a reference signal?

I have code working for 4k Extreme, but the Duo 2 setup is a little different. It requires setting up SD1 for full duplex. At that point, SD1 should become the key signal and SD2 should become the fill. I don't have reference, but that is not required AFAIK.

Yes. I was advised by support that I encountered a bug in the 10.9.0 driver. I haven't tried the newest 10.9.X driver which may or may not have a fix. For the time being though, I was also told that the 10.8.5 driver does not have this issue with the Duo 2. I haven't dried the Duo 2 with the driver yet, but I plan to.