Log Archiver Configuration

The Triton Log Archiver uses Hermes to archive logs. After set up, some configuration is necessary. The configuration files for Hermes can be found in the /opt/smartdc/hermes/etc/ directory.

Hermes configuration

When the headnode is configured, much of the data required for automatic configuration of the Hermes instance found in the logarchiver0 zone is already present. However, if Hermes is not used in the sdc zone, additional configuration will be required.

Viewing the current configuration

To view the current configuration:

1. From the headnode, run sdc-login to access the logarchiver0 zone:

[root@headnode (coal) ~]# sdc-login logarchiver0
[Connected to zone 'ac0cdcc4-2596-493c-9ece-c4a11be29f7e' pts/2]
Last login: Thu Jun 27 17:42:53 on pts/2
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (minimal-multiarch-lts 15.4.1)
                   `-'  https://docs.tritondatacenter.com/images/smartos/minimal

[root@ac0cdcc4-2596-493c-9ece-c4a11be29f7e (coal:logarchiver0) ~]#

2. Log in to the logarchiver0 zone and view the config file from /opt/smartdc/hermes/etc/config.json.

The current configuration looks similar to the following example:

[root@ac0cdcc4-2596-493c-9ece-c4a11be29f7e (coal:logarchiver0) ~]# PS1=${PS1//\\u@\\h /}
[(coal:logarchiver0) ~]# cat /opt/smartdc/hermes/etc/config.json
{
  "admin_ip": "10.99.99.38",
  "port": 9999,
  "log_level": "",
  "sapi": {
    "url": "http://sapi.coal.local"
  },
  "vmapi": {
    "url": "http://vmapi.coal.local"
  },
  "cnapi": {
    "url": "http://cnapi.coal.local"
  },
  "mahi": {
    "url": "http://mahi.coal.local"
  },
  "manta": {
    "user": "",
    "url": "",
    "key_id": "22:fb:86:4d:44:a2:ff:d5:63:06:4a:ff:08:1f:94:ab",
    "connect_timeout": ""
  },
  "max_concurrent_bootstraps": 32,
  "service_name": "logarchiver"
}

3. Check whether the Manta user and URL are configured. At this point, you will need to configure them.

  "manta": {
    "user": "",
    "url": "",
    "key_id": "22:fb:86:4d:44:a2:ff:d5:63:06:4a:ff:08:1f:94:ab",
    "connect_timeout": ""
  }

Configuring Manta user and URL

To configure the SDC_MANTA_USER and SDC_MANTA_URL on the sdc application, the account used as SDC_MANTA_USER must have write access to the log file archive locations in Manta locations. Before running these commands, make sure to:

• Run the commands on the headnode.
• Replace admin and https://us-central.manta.mnx.io with the username and URL that are appropriate for your environment.

Important: This may affect other services running in the sdc zone.

1. Run sapiadm update:

sapiadm update $(sdc-sapi /applications?name=sdc | json -H 0.uuid) \
    metadata.SDC_MANTA_USER=admin
sapiadm update $(sdc-sapi /applications?name=sdc | json -H 0.uuid) \
    metadata.SDC_MANTA_URL=https://us-central.manta.mnx.io

In a short while, the configuration automatically propagates to a Hermes config.json.

2. To force an immediate update, restart config-agent in the logarchiver0 zone:

[root@headnode (coal) ~]# sdc-login logarchiver0
...
[root@ac0cdcc4-2596-493c-9ece-c4a11be29f7e (coal:logarchiver0) ~]# PS1=${PS1//\\u@\\h /}
[(coal:logarchiver0) ~]# svcadm restart config-agent
[(coal:logarchiver0) ~]# cat /opt/smartdc/hermes/etc/config.json | json manta
{
  "user": "admin",
  "url": "https://us-central.manta.mnx.io",
  "key_id": "22:fb:86:4d:44:a2:ff:d5:63:06:4a:ff:08:1f:94:ab",
  "connect_timeout": ""
}

Changing the Manta SSH key

If the key_id shown in config.json does not correspond to an appropriate key for the chosen Manta user, the key will need to be updated.

NOTICE: This may affect other services running in the sdc zone.

1. Get the private and public SSH key files to the headnode global zone.
2. Run the following update script. The script assumes that the private key is in /var/tmp/mykey.id_rsa and the public key is in /var/tmp/mykey.id_rsa.pub. In this case the admin user is modified, matching the admin user from the previous section:

#! /bin/bash -ex

# Update the 'sdc' SAPI service
keypath=/var/tmp/mykey.id_rsa
keyid=$(ssh-keygen -l -f "$keypath.pub" | awk '{print $2}')
/usr/node/bin/node -e "
    var fs = require('fs');
    var d = {
        metadata: {
            SDC_PRIVATE_KEY: fs.readFileSync('$keypath', 'ascii'),
            SDC_PUBLIC_KEY: fs.readFileSync('$keypath.pub', 'ascii'),
            SDC_KEY_ID: '$keyid'
        }
    };
    console.log(JSON.stringify(d,null,2));
    " >/var/tmp/sdc-key-update.json
sdc_app=$(sdc-sapi /applications?name=sdc | json -Ha uuid)
sapiadm update "$sdc_app" -f /var/tmp/sdc-key-update.json
rm -f /var/tmp/sdc-key-update.json

# Update the key on the 'admin' user.
datacenter_name=$(bash /lib/sdc/config.sh -json | json datacenter_name)
sdc-useradm delete-key admin "$datacenter_name sdc key" || true
sdc-useradm add-key -n "$datacenter_name sdc key" admin "$keypath.pub"

Add external network

If the Manta URL configured above is not accessible from the admin network, the logarchiver0 zone will require an additional NIC. The following illustrates how to add the NIC from the command line. Alternatively, the Adding a NIC procedure may be used with the Operations Portal:

[root@headnode (coal) ~]# sdc-vmapi /vms/$(vmadm lookup -1 alias=logarchiver0)?action=add_nics -X POST -d@- <<EOP | sdc sdc-waitforjob
{
"networks": [{"primary": true, "name": "external"}]
}
EOP
HTTP/1.1 202 Accepted
Connection: close
workflow-api: http://workflow.coal.joyent.us
Content-Type: application/json
Content-Length: 100
Date: Tue, 25 Jun 2019 18:55:06 GMT
Server: VMAPI/9.8.11
x-request-id: 1151718a-b9af-4be7-9bd4-175aeee755af
x-response-time: 1095
x-server-name: 4dfff012-45f0-44b7-a23d-7d26bb630a4e

{
  "vm_uuid": "eb394c52-916e-4b95-aa53-3a7b6452824d",
  "job_uuid": "1ecffc08-92e0-4bbf-a094-e7710bc172cf"
}

Wait for job 1ecffc08-92e0-4bbf-a094-e7710bc172cf to complete (timeout=none)
..........
Job 1ecffc08-92e0-4bbf-a094-e7710bc172cf succeeded (elapsed=23s)
[root@headnode (coal) ~]# sleep 10  # wait for the logarchiver0 zone to reboot

Log Set Configuration

The set of logs that are to be archived are configured in /opt/smartdc/hermes/etc/logsets.json. The file consists of a single JSON array of objects. Each of the objects in the array represents one log set. For example:

[
  {
    "name": "hagfish_logs",
    "search_dirs": [ "/var/log/usage" ],
    "regex": "^/var/log/usage/([0-9]+)-([0-9]+)-([0-9]+)T([0-9]+)\\.log\\.gz$",
    "manta_path": "/%u/stor/sdc/usage/%d/%n/#y/#m/#d/#H.log.gz",
    "debounce_time": 600,
    "retain_time": 21600,
    "date_string": {
      "y": "$1", "m": "$2", "d": "$3",
      "H": "$4"
    },
    "zones": [
      "global"
    ]
  },

  {
    ...
  }
]

More examples can be found in the Examples page.

Each log set is made up of the key-value pairs that are described below:

Key Name	Type	Description
name	string	A unique identifier for this log set.
search_dirs	array of strings	A list of directories to search for matching logs. Must not be used with search_dirs_patterns.
search_dirs_pattern	string	A shell glob that will match the names of directories to search. Must not be used with search_dirs.
regex	string	A regular expression that should match files in the search_dirs or search_dirs_pattern. Back references in this pattern may be used in date_string and customer_uuid.
manta_path	string	The path to which each matching log file will be stored in Manta. This path may use the substitutions described below.
debounce_time	number	The number of seconds since a file's last modification or the time parsed from its name before it will be handled by the log archiver. The default is 600 seconds.
retain_time	number	The minimum number of seconds that a file is to be retained on the compute node after being archived. The default is 0 seconds. Set to value greater than 0 to retain recently archived files on the compute node.
date_string	object	This object has keys y (year), m (month), d (day of month), H (hour), M (minute), and S (second). Values are back references from regex in the form "$1". The defaults for unspecified values are {"y": "0000", "m": "01", "d": "01", "H": "00", "M": "00", "S": "00"}.
date_adjustment	string	The time obtained from regex and date_string may be adjusted before the time-related substitutions are used with manta_path. This value is of the form "[-]number[H]". For example, if a log file is rotated each day and log rotation program names the file with the rotation time, it may be desirable to store the file in Manta with the time the log file started. In this case a value of "-24H" would be used. If H is not specified, the adjustment is in seconds.
no_upload	boolean	Do not upload the file, just delete it after retain_time plus debounce_time has passed.
customer_uuid	string	Use the specified back reference from regex as the customer UUID. The customer UUID is then translated into a customer name and usable with the %U substitution.
zones	array of strings	A list of zone roles to which this rule applies. Use "global" for global zones.

In the manta_path key, the following substitutions are allowed:

Substitution	Description
#y	The year, as obtained from regex and date_string and modified by date_adjustment
#m	The month, as obtained from regex and date_string and modified by date_adjustment
#d	The day of month, as obtained from regex and date_string and modified by date_adjustment
#H	The hour, as obtained from regex and date_string and modified by date_adjustment
#M	The minute, as obtained from regex and date_string and modified by date_adjustment
#S	The second, as obtained from regex and date_string and modified by date_adjustment
%d	The datacenter name
%n	The node name (or zone name if a zone)
%u	The manta username from manta.user in config.json
%U	The customer username, obtained via customer_uuid
%z	The zone name
%r	The zone role ("workflow", "adminui", etc.)
$1, $2, ...	Back references from regex