Triton networking layout

Modified: 09 Mar 2023 22:10 UTC

This document describes the minimum networking requirements for running Triton and provides guidance on sizing these networks. This is a high level overview of the Triton networking layout and provides context for the detailed data gathered in the in the Triton network configuration document.

Physical networking and data cabling wiring

Each server will need its Serial-Over-IP (IPMI) connector and its NICs cabled to the site's local network wiring. All servers must be connected to core networking via one or more Top of Rack Switches (ToRS).

Network summary

Network Used By Used For Notes
Admin All servers (head node and compute nodes), core services Orchestration and management of Triton Needs to be untagged or native VLAN on the switch; cannot share interface with any other networks; must not have Internet access. A 1Gb connection is sufficient for this traffic.
External Internet facing core services, Internet facing containers Access to the Internet Can either be for Triton use only or can be used for end-user containers; can share interface with other traffic.
Underlay Compute nodes Fabric networking (VXLan) Should not have Internet access; can share interface with other traffic. Jumbo frames (MTU 9000) are required for the underlay; the use of a MTU less than 9000 on the underlay network is not supported.

Pool summary

Network Pool Used By Used For Notes
NAT Pool NAT Zones in Fabric networks Internet access A network pool is required for use by the NAT zones in Fabric Networks. The pool must contain at least one network, which has outbound internet access. "External" is often included in this pool. This can be an existing network; it can also be a collection of networks.

Required networks

Triton relies on having three (3) subnets and corresponding VLANs configured prior to installing Triton. Admin and External are the initial networks referenced in the config file and must be present and functional at initial install time. Additional networks can also be created, based on the desired configuration.

NOTE: some users have demonstrated that, given sufficient effort, they can install Triton without separate VLANs or separate NICs for the required networks. While we applaud their efforts, such topologies are not (and will not be) supported.

Additionally, the process of enabling fabrics (VXLan, or software-defined networking) requires that the Underlay network to be configured and functional. This network requires Jumbo Frames (MTU 9000). For more information, please see the Triton networking and fabric operations guide. Triton does not support changes to network or NIC Tag MTUs on the underlay network post-installation; the underlay network must be properly configured prior to installation.

To enable NAT from user fabric networks you must create a NAT Pool, which is comprised of 1 to n networks. By default, this can use the External network; however, it is possible to create and use a different L2/L3 network for this pool provided it has Internet access. It should be noted that it is possible to add/remove networks from this NAT Pool post-setup. Additionally, it is possible to disable this functionality if it is not needed, although a NAT Pool will still need to be defined in the configuration.

Any additional networks - both L2 and L3 - can be configured/added following the completion of the Triton install procesinstall process. Please note that MNX recommends that a separate network be used for remote access to the hardware management ports. All networks used by Triton must be dedicated, and contain no additional hardware other than switches and routers.

Firewall rules

Both the Admin and the Underlay network must be free of firewall rules. These networks must not have Internet access, and are only used internally by Triton.

The External network requires, at a minimum, outbound access to the Internet via the following ports for all core service zones as well as the head node itself:

In the event local security policies prohibit direct Internet access, Triton supports the use of proxies. However, you will need access to local DNS and NTP services in order to install and operate Triton. Please contact MNX support if you have any questions regarding these requirements.

Note that if you are using the External network for end-user containers, you will most likely want to allow full access (inbound and outbound) for the addresses used for end-user containers.

Triton supports Link Aggregation via the LACP protocol, provided that the TORS being used supports a "LACP Fallback" mode to allow the compute nodes to PXE boot. Please contact your switch manufacturer in order to confirm that your switch meets these requirements.

Network detail