USB hub disconnects and fails to re-enumerate

On a custom board I'm working on I'm running into a periodic issue with a set of cascaded USB hubs. What I'm seeing is that occasionally one of the hubs (hub 3 in the attached schematic) will disconnect, start to reconnect, and then fail to negotiate current draw. This same scheme worked fine on an AM3352 based board in the past. I've tried manually toggling the reset line (which is mapped to GPIO4_20), but I wasn't able to replicate the same behavior. The disconnect would happen, but the hubs would reconnect normally.

For some extra information, the hub outputs are connected to FTDI USB to serial converters, and add up to a total of 20 serial ports. When we access said serial ports, we only keep 1 file handle open at a time to avoid overloading the bus.