One bit Alpha channel transparency (DTX1) will always appear with rough edges, so use DTX3. I believe you have an alpha channel in your dds file? In Xpacker for material properties set for Transparency "Use Alpha" or "Use Chroma". I'm not really sure if there's any difference for DTX 3 (maybe Chroma takes less resources to calculate, and I heard that rFactor liked Chroma better). If you use interpolated Alpha (DTX 5), it must be "Use Alpha" - for e.g gradient transparency from 0 to 100%.
It does not work like you have just white background on the image and you think when you select white color for "Use Chroma" - you can expect it will be transparent. You must have an Alpha channel. More - for dark fence like yours I would suggest to make the background dark and not white - this will eliminate potential light egdes when the alpha channel in not very sharp. In PS and I believe in other programs, you can correct alpha channel (edge sharpness) with channels, curves, paint, erase - just standart PS's operations when working on pictures. So, the black color on alpha channel represents transparency, the white is what you will see. And then the Chroma will treat the white color as visible (it's again about alpha channel, not the image).

On DTX 1: The original RBR opaque textures (roads and terrain) have 1 bit alpha, so they are dtx1. You don't need the alpha channel in such textures for RX tracks (bigger files) - I think the creators of the game had some tools for using transparency "on the fly" - like smooth texture crossovers or something like that (because there are not textures just painted that way).