Setting up Triton virtual networking and fabrics

Modified: 17 Jan 2023 21:28 UTC

To set up and enable fabrics in an instance of Triton, there are several steps that must be taken. First, the design of the underlay network needs to be completed. Second, various hardware and software prerequisites must be met and verified. Once those tasks are complete, fabrics can be enabled and deployed to the compute nodes.


The primary decision that must be made is the design of the underlay network, specifically the physical design. The one hard requirement with the underlay network is that it must have jumbo frames (MTU 9000) enabled; running without jumbo frames enabled is not a supported configuration.


There are three phases of prerequisite work that must be completed before beginning the set up of fabric networks.

Note: the network hardware prerequisite work can go on at the same time as the other work.

Network hardware prerequisites

The first thing that should be done is performing any necessary maintenance to network hardware, including, but not limited to, switches and routers. As part of this, any necessary VLANs should be configured for the underlay network and configuration changes to enable jumbo frames should be made.

Triton software update

The Triton software in use should be updated to the latest version before beginning. This will ensure that all of the necessary software is up to date before proceeding to enable fabric support. Customers with current support contracts should contact MNX support for assistance with this step.

Compute node updates

Triton uses iPXE to PXE boot compute nodes; as part of using fabrics, there is a required update to iPXE for all compute nodes. If a compute node is not booting via iPXE based on a USB key, then the first step is to transition to that. Once that is taken care of, you can use sdcadm to ensure that iPXE is up to date everywhere. Note: It is possible to boot using the onboard PXE on your hardware; however, it is highly recommended that you transition to using USB keys in all of your compute nodes in order to streamline the process and take advantage of additional features available in the customized iPXE loader.

You can take care of this by running:

headnode# sdc-oneachnode -a "sdc-usbkey update"

Note: This upgrade will not require a reboot to take place and will not interrupt any system services.

Deploying fabrics

Please see the Triton Advanced Configuration page for the steps required to deploy fabrics, including detail on the creation of NIC Tags, networks, and other configuration values.

Note: The process of deploying fabrics requires a reboot of all compute nodes that are participating in the fabric network. There may be situations where such reboots are scheduled over a long period of time. In that case, the recommendation is that all compute nodes be set up, stopping short of the reboot. This provides the easiest upgrade path for these compute nodes to be brought up as part of the fabric network when they can be rebooted.