APIs

Triton networking layout

Modified: 22 Nov 2017 17:37 UTC

This document describes the minimum networking requirements for running Triton and provides guidance on sizing these networks. This is a high level overview of the Triton networking layout and provides context for the detailed data gathered in the in the Triton network configuration document.

Each server will need its Serial-Over-IP (IPMI) connector and its NICs cabled to the site's local network wiring. All servers must be connected to core networking via one or more Top of Rack Switches (ToRS).

Needs to be untagged or native VLAN on the switch; cannot share interface with any other networks; must not have Internet access. A 1Gb connection is sufficient for this traffic.

External

Internet facing core services, Internet facing containers

Access to the Internet

Can either be for Triton use only or can be used for end-user containers; can share interface with other traffic.

Underlay

Compute nodes

Fabric networking (VXLan)

Should not have Internet access; can share interface with other traffic. Jumbo frames (MTU 9000) are required for the underlay; the use of a MTU less than 9000 on the underlay network is not supported.

A network pool is required for use by the NAT zones in Fabric Networks. The pool must contain at least one network, which has outbound internet access. "External" is often included in this pool. This can be an existing network; it can also be a collection of networks.

Triton relies on having three (3) subnets and corresponding VLANs configured prior to installing Triton. Admin and External are the initial networks referenced in the config file and must be present and functional at initial install time. Additional networks can also be created, based on the desired configuration.

NOTE: some users have demonstrated that, given sufficient effort, they can install Triton without separate VLANs or separate NICs for the required networks. While we applaud their efforts, such topologies are not (and will not be) supported.

Additionally, the process of enabling fabrics (VXLan, or software-defined networking) requires that the Underlay network to be configured and functional. This network requires Jumbo Frames (MTU 9000). For more information, please see the Triton networking and fabric operations guide. Triton does not support changes to network or NIC Tag MTUs on the underlay network post-installation; the underlay network must be properly configured prior to installation.

To enable NAT from user fabric networks you must create a NAT Pool, which is comprised of 1 to n networks. By default, this can use the External network; however, it is possible to create and use a different L2/L3 network for this pool provided it has Internet access. It should be noted that it is possible to add/remove networks from this NAT Pool post-setup. Additionally, it is possible to disable this functionality if it is not needed, although a NAT Pool will still need to be defined in the configuration.

Any additional networks - both L2 and L3 - can be configured/added following the completion of the Triton install procesinstall process. Please note that Joyent recommends that a separate network be used for remote access to the hardware management ports. All networks used by Triton must be dedicated, and contain no additional hardware other than switches and routers.

Both the Admin and the Underlay network must be free of firewall rules. These networks must not have Internet access, and are only used internally by Triton.

The External network requires, at a minimum, outbound access to the Internet via the following ports for all core service zones as well as the head node itself:

NTP (Port 123)

DNS (Port 53)

HTTP (Port 80)

HTTPS (Port 443)

HTTP Alternate (Port 8080)

In the event local security policies prohibit direct Internet access, Triton supports the use of proxies. However, you will need access to local DNS and NTP services in order to install and operate Triton. Please contact Joyent Support if you have any questions regarding these requirements.

Note that if you are using the External network for end-user containers, you will most likely want to allow full access (inbound and outbound) for the addresses used for end-user containers.

Triton supports Link Aggregation via the LACP protocol, provided that the TORS being used supports a "LACP Fallback" mode to allow the compute nodes to PXE boot. Please contact your switch manufacturer in order to confirm that your switch meets these requirements.