Wire up the CloudLab2 user allocatable switches

We need a plan for how the user allocatable switches will be connected to the netscouts and how they might be hardwired to each other. Some of the issues and ideas in no particular order:

Do we connect the netscouts to the regular experiment fabric? This would allow us to have a second experimental interface on nodes. One scenario where this is useful came up lately where an experimenter wanted to have a blockstore mounted but also wanted low-level control (DPDK) over the experimental interface. Currently, a blockstore has to share the same physical link as any topology setup by the user.

The nodes are distributed across the two netscout switches and the two netscouts are not interconnected. Thus to access an arbitrary node from any user alloc switch, the user alloc switches will have to be connected to both netscouts.

Do we want direct wires between the user alloc switches? While we can interconnect the switches via the netscouts, that limits the interconnects to 40Gb. We could directly connect the Mellanoxes in particular at 100Gb allowing for example a multi-level fat tree topology.

What do we envision the common user alloc switch topos to be? Do we emphasize more, simple (i.e., single switch) simultaneous topologies over fewer, more complex (e.g., multiple level, multiple switch) topologies? This is largely related to the previous bullet--the only way we are really going to be able to form complex topologies will be to take advantage of the extra "uplink" (40Gb on Dells, 100Gb on Mellanox) ports directly wired to each other.

Obviously, the plan will evolve over time and any plan will not survive first contact with experimenters. For the milestone of getting CloudLab2 resources operational, we need to decide on our initial plan.

Notes:

Each netscout has 3 blades of 24 40Gb ports each, for a total of 72 40Gb or up to 288 10Gb per switch. So 144 40Gb or 576 10Gb ports. Presumably we can mix and match, with some ports broken out at 10Gb and others not. So we should have plenty of flexibility.

Six of the 24 ports on each netscout are "Smart Ports". We don't yet know the implications of this, in particular, whether it limits what we can do with those ports.