I'm using structure synth to create a fractal. So I want to put map tetrahedrons onto the 4 face's of a larger tetrahedron. In sturcture synth you only have option of using a 3x3 transformation matrix though...

Actually, most 3d drawing software allow affine transformations. They are needed for translations and also perspective projections.

My suggestion is for a fractal, the tetrahedron should at least be similar (which is not the case for your example).
If they are similar, you can easily draw it with a scaling, rotation, and translation (which is an affine transformation).
Also, perhaps it is easier to use the scaling rotation and translation primitive instead of constructing the matrix yourself (essentially let the computer do it for you).

I drew that picture by explicitly stating the coordinates of both tetrahedra to the program. However I want the program to build the second by reiterating the first and applying a transformation, as this method will allow me to add levels of tetrahedra arbitrarily deep into the figure...

I've tried to solve the system of equations by transforming each vertex on one tetrahedron into the vertex on another, as (for example) follows: