Very difficult permutation

There is an array of 16 squares, arranged in a 4 x 4 grid. A supply of 4 A's, B's, C's, and D's are given. How many distinguishable ways are there of placing each of the letters in a square, if, one letter must appear once in each row and each column?

I'm lost. There are 4! ways of placing the letters in the first row. But then when you get to the second row, you can place the A, which has 3 possible locations. But then the B, depends on where the A was placed. If the A was placed under the B in row 1, then you have 3 possible spots left, but if the A was placed under the C, then there are only 2 spots left. Shown below.

The letters represent their placement, the 0 represents a spot that a B can't be placed, and a - represents a spot the B CAN be placed.
R1: ABCD
R2: -A--

or

R1: ABCD
R2: -0A-

See the problem? How do I get around this????:surprised :surprised :surprised

A can be placed in any of the four rows in the first column. In the second row there are three possible places for A while in the third row there are two and if all these are once fixed there is just one fixed position for A in the fourth row. In any such possibilities, B can be placed in the first column in any of the three rows where b is not kept. In the second row we similarly have two likely positions for B since 1 position is avoided by the existence of b in that row in the first column while the other is restricted due to the presence of the A in that square. Then the other positions of B are fixed. Again there are two possible places for C in the first column and once this is fixed the other three placement of C is also fixed. Once the placement of these three alphabets is fixed then the placement of D is also fixed. Thus the number of possible cases as given above is 4*3*2*3*2*2 = 2^5*3^2.