Simplest solution would be to start with a SoC that has all the peripherals that you need, e.g. a SAMA5D35 or SAMA5D36 (but with less crypto capability).

The simplest add-on would be a USB-Ethernet adapter; apparently it's "robust" enough for the RaspberryPi.
If you have (or can buy) the expertise, a discrete EMAC+PHY could be connected to the EBI of the SoC.

Ethernet modules with UART or SPI interfaces would only be suitable if it connected as a network device rather than as a serial modem-like device.

If you're running Linux, you can use virtual LANs (VLAN) on the only Ethernet as the 802.1Q trunk port. The Linux driver understands the virtual ports and can keep the data flows separate.

To separate the virtual circuits to real ones, you need to replace the PHY with a managed Ethernet switch chip, e.g. KS8995 with one port to the MAC and 4 ports to the outside. This is how the small firewall/router boxes handle the ports.

Another possibility is to have a managed switch with VLAN capabilities and use it to handle the only Ethernet from the processor. I built a router / firewall from a Raspberry Pi trunked to an old 3COM managed switch.