Triton Container Name Service

Modified: 05 Nov 2024 18:47 UTC

This document will walk you through the process of planning, installing, and configuring Triton Container Name Service for several common use cases. It also provides general advice on customization options. This is not an exhaustive document in that there are additional configuration options available beyond what is covered here, but the most common options and configuration values are covered.

Introduction

Container Name Service is flexible in it's configuraion options in order to accomodate the many different forms of DNS deployments. This means that unlike many of the other Triton services, there is no "standard way" to deploy CNS. The service is designed to integrate into an existing DNS infrastructure if one is available, or function as a standalone server if it is solely to be used to resolve DNS names managed by CNS.

CNS components

CNS is deployed as an additional core service (cns) running in it's own zone, by default on the Headnode. The zone runs two services, the cns-updater, and the cns-server. The cns-updater is responsible for gathering data from all of the other APIs and parts of SDC and turning it into DNS records. The cns-server is responsible for serving these DNS records to clients (and other nameservers).

The cns-server also provides a small REST API that can be used to inspect the state of the overall system. This API can be interrogated using the the cnsadm tool inside the CNS zone.

Configuration data for the CNS system is stored in the SAPI metadata for the CNS service, and can be edited using the cnsadm command. This command has integrated help and also handles data validation to ensure that any resulting configuration changes are valid.

Outside the Triton deployment, there are components of your DNS infrastructure that can also participate in the CNS system. Any nameserver that can perform standard DNS zone transfers (AXFR/IXFR) can be a secondary nameserver for CNS and replicate all of its records. The use of secondary nameservers are key to providing availability and scalability with CNS.

How clients look up names in CNS

One very important thing to consider and fully understand before CNS deployment is how client machines will look up names from CNS.

In a CNS deployment, "client machines" may mean just the machines on the local intranet (for internal use only), or might mean any machine on the Internet (for public use).

When a client machine goes to look up a name in DNS, it will have a local configuration file (typically /etc/resolv.conf), which specifies a number of recursive nameservers to be used to perform lookups. Often these are servers provided by an ISP or some local cache.

To use CNS in a restricted corporate intranet setting where the records are only resolvable from inside the local network, the recursive nameservers used by all client machines will need to be configured in order to be able to find CNS directly.

If clients are using public recursive nameservers, or ones belonging to an ISP, then they will obey only the DNS resolution rules for the public Internet, meaning that CNS records need to be available to the entire Internet. This will require that delegation and glue records are setup correctly so that these recursive nameservers know how to find your CNS.

CNS is designed to primarily function as an authoritative nameserver, rather than a recursive one. This means that clients do not directly query it for all of their DNS lookups, but instead are referred to it by their normal recursive nameserver, or the recursive nameserver makes the query on their behalf. However, it can be used in small deployments as a recursive nameserver that answers only for names under the configured CNS zone. It has been implemented to return a SERVFAIL error for names outside its designated suffix, so that client implementations will assume there was an error and move on to the next nameserver in their resolver list. This mode of operation is only recommended for development and testing, due the the number of possible failure modes in the face of network interruptions and downtime of the CNS zone.

DNS Zones

To better understand how CNS works, you must understand DNS zones. A DNS zone is a portion of the domain name space in the Domain Name System which spells out the rules for a specific domain (and only that domain).

A DNS zone is a sub-tree of the DNS hierarchy -- e.g. you could own the domain example.com, and use the zone cns.example.com for CNS -- so your actual CNS hostnames would appear as foobar.cns.example.com.

Since containers and instances in Triton may have NICs on multiple networks (and therefore multiple IP addresses), it is often useful to distinguish between them. If you deployed CNS with a single DNS zone in use, in the default configuration, you would find that looking up a CNS name returns to you all of the IP addresses of a container mixed together as one list, regardless of whether some are "private" or "public" in your deployment. This may be undesirable if you expect users to connect only to some of these addresses (because, for example, some may not be accessible from the outside).

CNS supports the use of multiple DNS zones to make this distinction clear. The most typical use is to have one DNS zone for "public" IP addresses, and one for "private" IP addresses. It is worth noting that these do not have to be actual private IP addresses in terms of RFC1918 (e.g. in the subnets 10.0.0.0/8, 192.168.0.0/16 etc) -- this is about the semantic use of the network in your deployment.

DNS zones handled by CNS can either be configured to list IP addresses only from a defined set of networks (or network pools), or can be configured as "catch-all" or "wildcard" zones, which list all remaining addresses. The most typical configurations are to list "only public IP addresses", and to list "public" and "private" addresses in two separate zones (often the "private" zone is made a catch-all zone so it can list Fabric networks).

Example deployment designs

Small development/testing setup

Internal-only corporate setup

Existing infrastructure before CNS:

CNS deployment:

Publically resolvable setup

Existing infrastructure before CNS:

CNS deployment:

In this setup, all CNS generated names are resolvable from the public Internet.

Hybrid setup

It is possible to have a hybrid of the "publically resolvable" and "internal-only corporate" designs from above, as CNS supports multiple DNS zones at once where each zone contains only containers/VMs that have NICs on some particular subset of your SDC networks. This can be useful in order to make some "public" subset of your containers visible to the outside world whilst keeping internal interfaces and addresses private.

To achieve this configuration, both designs at once from the two previous sections are setup, but only the "public" zone is configured in hidden master mode. The combination of the two works because each set of nameservers (the public Internet-facing set, and the internal recursive set) only replicate their particular zone from the CNS server.

Tasks

Setting up CNS for the first time

Entering the CNS zone and viewing configuration

To enter the CNS zone and view the current configuration of the system:

headnode# sdc-login cns
(cns)# cnsadm config
my_name:         cns.dc.example.com
hostmaster:      hostmaster@example.com
use_login:       false
use_alias:       true
allow_transfer:  127.0.0.1
(cns)# cnsadm zones
ZONE                     NETWORKS   PEERS                      HIDDEN PRIMARY
dc.cns.example.com       *                                     false
(ip-reverse-lookup)                                            false

These two commands - cnsadm config and cnsadm zones - give a basic overview of how CNS is currently configured. The above example output shows the default configuration that will appear after creating a new CNS zone in a datacenter called dc with a DNS suffix of example.com. Currently this CNS will generate records under the DNS zone dc.cns.example.com for all enabled VMs in the datacenter on all networks (indicated by the *). It is not set up to allow any replication peers, and is not configured as a Hidden Primary.

An example instance record in this zone could look like example.inst.6bfa28b6-e64c-11e5-adf5-5703f12edb00.dc.cns.example.com (the zone name is the suffix appended after the user UUID).

The information presented in cnsadm config is used across all DNS zones served by CNS. The first two fields, my_name and hostmaster determine the information that appears in SOA (start-of-authority) records. These records identify metadata about a DNS zone and its management.

The fields use_login and use_alias determine whether VM/container aliases (short names) and user logins will be used in DNS names. By default, container aliases are used, and user logins are not (this is the configuration deployed in the MNX Public Cloud). CNS will always generate records corresponding to the UUIDs of containers and users -- these flags only determine whether to additionally generate records with shorter friendly names.

For example, if use_login was enabled, the example instance record mentioned above could also be found under the name example.inst.fred.dc.cns.example.com, given that the owner's login username is fred.

The allow_transfer field contains a list of IP addresses or CIDR-format subnet masks that should be allowed to become replication peers. Note that this has some overlap with the "peers" property on a particular zone.

You can also view detailed configuration about one zone using cnsadm:

(cns)# cnsadm zones dc.cns.example.com
zone:            dc.cns.example.com
networks:        *
peers:           []
hidden_primary:  false

This is particularly useful if the information was truncated in the table summary display, as will often happen when network UUIDs are explicitly listed under networks, or more than one replication peer is used.

Configuring DNS zones

From the CNS zone on the headnode, you can use the cnsadm zones command to manage DNS zones in the CNS configuration. This is the output of cnsadm zones with no arguments, for a typical default configuration:

[root@uuid (dc:cns0) ~]# cnsadm zones
ZONE                     NETWORKS   PEERS                      HIDDEN PRIMARY
dc.cns.example.com       *                                     false
(ip-reverse-lookup)                                            false

Here we have a single DNS zone, dc.cns.example.com, configured as a catch-all or wildcard zone (indicated by * under NETWORKS).

To change this to a "public IP addresses only" configuration, we would simply modify the network list on the zone:

[root@uuid (dc:cns0) ~]# cnsadm zones dc.cns.example.com networks=8c26b4f8-b67e-11e6-8ee4-ffb3a2f73c8d

8c26b4f8-b67e-11e6-8ee4-ffb3a2f73c8d is the UUID of the "external" network on this deployment -- you can obtain this UUID from the Networking tab in AdminUI, or by using the sdc-napi command.

This would change the output of cnsadm zones to now look like:

[root@uuid (dc:cns0) ~]# cnsadm zones
ZONE                     NETWORKS   PEERS                      HIDDEN PRIMARY
dc.cns.example.com       (1 UUIDs)                             false
(ip-reverse-lookup)                                            false

Now, if we wanted to change to a public-private split configuration, we would add a second zone as a new wildcard:

[root@uuid (dc:cns0) ~]# cnsadm zones -a dc-int.cns.example.com networks=*

And the new output of cnsadm zones:

[root@uuid (dc:cns0) ~]# cnsadm zones
ZONE                       NETWORKS   PEERS                      HIDDEN PRIMARY
dc.cns.example.com         (1 UUIDs)                             false
dc-int.cns.example.com     *                                     false
(ip-reverse-lookup)                                              false

The man reference page about the cnsadm command includes further examples of modifying, adding and removing DNS zones. Type man cnsadm while logged into the CNS zone for further details.

Checking CNS status

Enter the CNS zone from the headnode, and check that no services are down or in maintenance:

headnode# sdc-login cns
(cns)# svcs -xv

If svcs -xv produces no output, then all services are running.

If it does produce output, take a look in the service logs for any malfunctioning services. In particular, the two SMF services cns-updater and cns-server are relevant. You can use a command like tail -n 500 $(svcs -L cns-updater) | bunyan to view nicely formatted logs from the cns-updater service.

The logs should give hints as to the source of your trouble, but it is likely if you reach this point that you have encountered a bug. Customers with active support contracts should include these logs and information about their current CNS configuration in a help ticket submitted to MNX support.

If services are running normally, use the cnsadm status command to check last changed times, serial numbers and the status of replication peers:

(cns)# cnsadm status
ZONE                       LATEST SERIAL  CHANGED
staging-1.cns.example.com  373423966      3 days ago
3.26.172.in-addr.arpa      373178900      4 wks ago

PEER         ZONE                       LATEST SERIAL  DRIFT  VERSION
172.24.2.49  3.26.172.in-addr.arpa      373178900             ISC BIND 9.10.3-P3
             staging-1.cns.example.com  373423966

This output is from a real working configuration to show what the replication peer status output looks like.

Here we can see that this CNS is configured with the zone staging-1.cns.example.com, and has generated records for it, as there is a valid serial number given. Reverse-lookup records have also been generated for IP addresses under 172.26.3.x.

This CNS currently has 1 known replication peer, 172.24.2.49, which has replicated both zones from it. We can see that the latest serial the peer has copied from us is the same as the latest serial generated. If this were not the case, there would be a note in the "DRIFT" column highlighting that this peer was behind.

We can also see, if available, the version of software running on the peer, to help with debugging.

The commandline tool dig can also be very valuable in debugging DNS-related problems. The tool is pre-installed in the CNS zone, as well as the SDC headnode. You can use it to look up a particular name for testing:

(cns)# dig example.inst.fred.dc.cns.example.com @localhost
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39569
;; flags: qr aa rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1200
;; QUESTION SECTION:
;example.inst.fred.dc.cns.example.com.        IN A
;; ANSWER SECTION:
example.inst.fred.dc.cns.example.com. 30 IN A 172.26.3.49

;; AUTHORITY SECTION:
dc.cns.example.com. 3600   IN      NS      cns.dc.example.com.
...

Here we can see that CNS returned the address 172.26.3.49 for this name. dig also displays very detailed information about the contents of the DNS packets exchanged with the server, which can help to point out problems.

Adding an ISC BIND server as a replication peer

Starting from the default configuration for dc.example.com shown above, we will proceed to set up an ISC BIND nameserver as a replication peer or "slave" (in BIND terminology) to serve CNS records.

First, as our BIND server is going to be placed on the external network, we will need to give the CNS zone an IP on that network as well to communicate with the BIND server.

headnode# /usbkey/scripts/add_external_nic.sh $(vmadm lookup alias=cns0)
headnode# vmadm get $(vmadm lookup alias=cns0) | json nics | json -a nic_tag ip
admin 10.0.0.107
external 10.0.1.82

Our existing BIND nameserver is running on 10.0.1.10, and is known by the DNS name ns1.example.com.

Add the nameserver as a replication peer in CNS:

headnode# sdc-login cns
(cns)# cnsadm config allow_transfer+=10.0.1.10
(cns)# cnsadm zones dc.cns.example.com peers+=ns1.example.com

And now add the following snippet into the BIND configuration file:

masters cns {
    10.0.1.82;
};

zone "dc.cns.example.com" {
    type slave;
    file "slave/dc.cns.example.com";
    masters { cns; };
};

Reload the configuration:

nameserver# rndc reload

And finally, check the output of cnsadm status to verify that the peer is now known and in sync:

(cns)# cnsadm status
ZONE                     LATEST SERIAL  CHANGED
dc.cns.example.com         373423966      1 minute ago
1.0.10.in-addr.arpa      373423966      1 minute ago

PEER         ZONE                     LATEST SERIAL  DRIFT  VERSION
10.0.1.10    1.0.10.in-addr.arpa      373423966             ISC BIND 9.10.2-P1
             dc.cns.example.com         373423966

Extra debugging information about replication

When investigating peer sync delays or other problems with replication, the cnsadm peers command can be of use:

(cns)# cnsadm peers 10.0.1.10
address:       10.0.1.10
version:       ISC BIND 9.10.2-P1
using_notify:  true
using_ixfr:    true
serials:
    1.0.10.in-addr.arpa: 373423966
    dc.cns.example.com: 373423966

counters:
    soa: 29
    ixfr: 12
    axfr: 4
    goodxfer: 8

This can show you whether a given peer is accepting NOTIFY commands, whether it is using IXFR (incremental transfers), and counters for errors and types of queries the peer has made.

The logs of the cns-server service can also be informative, as well as the logs of the peer nameserver itself.

As always, the dig command is very useful, particularly with its ability to request zone transfers (using dig axfr zone.name @localhost), which will show you the entire contents of a given server's version of a zone.

Further Detail

Additional detail on CNS, as well as the source code, is available on the GitHub triton-cns repository. As CNS is currently in heavy development, additional details and new features will be added to the repository before they are ported into this document.