From what I can tell, it's a rebadged RS/6000 320, with dual 160MB hard disks and four special MCA cards.
Three of the cards are Token Ring adapters, and the fourth is a dual serial card… All of the cards have Intel i960s and a RAM SIMM slot on them.
Also, it uses funky square serial ports, so I don't have any way of display what it's doing, apart from that diagnostic LCD.

I was able to find
this
Technical Manual, which I have yet to dig through.

The RAM card inside has 8 slots, all of which were filled when I got it, the two top slots were filled with two 16MB EDO RAM SIMMs, which I don't think work in these machines, and the other sticks were IBM "68X6271" 2MB SIMMs.

With the two EDO RAM sticks installed, the LCD display would stop changing at 213 (I believe) which is supposed to mean "bad memory board or processor planar", after removing the sticks (and shifting the rest of the SIMMs up) the LCD displays the following codes:

120 122 124 101 153 154 100 211 214 <flashing 888> [Disk spinup]

Now, after the 888 starts flashing, pressing the reset button cycles through the following codes

103 104 209 c01 100 200 300 413 500 600 700 800 <flashing 888>

Which apparently means something wrong with the RAM, I've shifted/swapped all the IBM 68X6271 RAM, and nothing has changed at all.

What's up here? What do I really have here, a router?
Is this even supposed to run AIX?

I have never owned a RS/6000, but from the 320H product sheet, it uses 10-pin serial ports (probably an AMP connector). There is an option #2936 that was included with the system, a cable betwixt the 10-pin "standard serial port" and DB-25. #2937 was the null-modem coupler for hooking up a serial terminal.
The owner's manual should describe which memory slots need to be filled, you will need to populate at least one complete bank.

Funny...I was just thinking of the RS/6000 routers a few days ago, and just today I thought about swinging by this forum after a long absence.

Anyway, the 6611 is indeed a router, and is a scaled down attempt by IBM at commercializing the RS/6000 routers used by the NSFnet from roughly 1990-1997. The original T3 based NSFnet used RS/6000-930s stuffed with those special Microchannel cards. These cards were somewhat experimental at the time. The big RS/6000s generally had three or four of these cards, one of which was almost always FDDI, and the rest being HSSI, ethernet, or V.35. These cards are what do the routing - the RS/6000s are just there for the ride. Those i960s run a (very) scaled down mutant AIX.

These routers were very good for their time, and the engineers really liked having a "real" Unix machine as a router. The router cards, however, ran out of steam around 1996, I think due to the routing tables being just too small. IBM chose not to do anything, so the T3 backbone (ANSnet by then) chucked the technology and replaced the RS/6000s with Bay and Cisco routers.

I think only the ethernet, token, and V.35 cards were released to the public, with the 6611 - I do not think FDDI or HSSI did. I likely have the software on many old tapes, to be dumped on Al in due time.

Don't try to turn the 6611 back into a 320 - you will have a bad time. I have not seen a 6611 in many years, but I think there are some gotchas.