Log Archiver Setup

Modified: 09 Mar 2023 22:10 UTC

Installation of the Triton Log Archiver involves creation of the logarchiver service and deployment of its agents. The Triton Log Archiver configuration is maintained in the logarchiver0 zone, which is also responsible for pushing agents and their configuration to compute nodes. Each logarchiver-agent uploads logs to the appropriate location in Manta by way of logarchiver-proxy in the logarchiver0 zone.

Requirements

The following are required to use Triton Log Archiver:

The logarchiver0 zone must be able to reach Triton Object Storage (Manta) instance.
Credentials must be available that allow write access to the appropriate locations in Manta.

Service Creation

The sdcadm post-setup logarchiver command is used to create the logarchiver service and its first instance, logarchiver0:

[root@headnode (coal) ~]# sdcadm post-setup logarchiver

This will make the following changes:
    create "logarchiver" service in SAPI
    download image a1b75ba0-336b-46f0-a9fb-3260947b2114 (logarchiver@master-20190507T165334Z-gb5754db)
        from updates server using channel "dev"
    create "logarchiver" service instance on server "564d2b36-7e25-7d79-e3c2-7ceb79d8abd6"

Would you like to continue? [y/N] y

Importing image a1b75ba0-336b-46f0-a9fb-3260947b2114 (logarchiver@master-20190507T165334Z-gb5754db)
Downloading image a1b75ba0-336b-46f0-a9fb-3260947b2114
    (logarchiver@master-20190507T165334Z-gb5754db)
Imported image a1b75ba0-336b-46f0-a9fb-3260947b2114
    (logarchiver@master-20190507T165334Z-gb5754db)
Creating "logarchiver" service
Creating "logarchiver" instance on server 564d2b36-7e25-7d79-e3c2-7ceb79d8abd6
Created VM eb394c52-916e-4b95-aa53-3a7b6452824d (logarchiver0)
Completed successfully (elapsed 109s).

Control over the server on which it is installed, which version to install, etc., are available via command line options. Use the --help option to find the help that is correct for the software version you are running:

[root@headnode (coal) ~]# sdcadm post-setup logarchiver --help

Create the "logarchiver" service and a first instance.

Usage:
     sdcadm post-setup logarchiver

Options:
    -h, --help                  Show this help.
    -y, --yes                   Answer yes to all confirmations.
    -n, --dry-run               Do a dry-run.
    -s SERVER, --server=SERVER  Either hostname or uuid of the server on which
                                to create the instance. (By default the headnode
                                will be used.).

  Image selection (by default latest image on default channel):
    -i ARG, --image=ARG         Specifies which image to use for the first
                                instance. Use "latest" (the default) for the
                                latest available on updates.tritondatacenter.com,
                                "current" for the latest image already in the
                                datacenter (if any), or an image UUID or
                                version.
    -C ARG, --channel=ARG       The updates.tritondatacenter.com channel from which to
                                fetch the image. See `sdcadm channel get` for
                                the default channel.

The "logarchiver" service uploads specific Triton log files to a configured Manta object store.

Agent Installation

Agents are installed on each compute node automatically by the svc:/smartdc/application/hermes:default SMF service running on the logarchiver0 zone.

The presence of the logarchiver-agent into a server can be verified using typical SMF commands:

root@computenode # svcs svc:/smartdc/logarchiver-agent:default
online         Jun_25    svc:/smartdc/logarchiver-agent:default

In case of missing logarchiver-agent instances on some servers, the reason can be detected by reviewing the hermes service logs within the logarchiver0 zone. For example, the following fragment of a log file details that the reason for the missing install on a given server that it's not running:

[2019-06-26T11:00:00.003Z]  INFO: hermes/27456 on eb394c52-916e-4b95-aa53-3a7b6452824d: deploying actor (server=564d7de2-18d1-3c68-f893-44008b349c7d)
[2019-06-26T11:00:00.017Z] ERROR: hermes/27456 on eb394c52-916e-4b95-aa53-3a7b6452824d: error deploying actor to server (server=564d7de2-18d1-3c68-f893-44008b349c7d)
    VError: could not execute command on server 564d7de2-18d1-3c68-f893-44008b349c7d: Operation attempted on server which is not running
        at /opt/smartdc/hermes/lib/servers.js:233:13
        at /opt/smartdc/hermes/node_modules/sdc-clients/lib/restifyclient.js:174:20
        at parseResponse (/opt/smartdc/hermes/node_modules/restify-clients/lib/JsonClient.js:93:9)
        at IncomingMessage.done (/opt/smartdc/hermes/node_modules/restify-clients/lib/StringClient.js:207:13)
        at IncomingMessage.g (events.js:292:16)
        at emitNone (events.js:91:20)
        at IncomingMessage.emit (events.js:185:7)
        at endReadableNT (_stream_readable.js:978:12)
        at _combinedTickCallback (internal/process/next_tick.js:80:11)
        at process._tickDomainCallback (internal/process/next_tick.js:128:9)

The location of the log file can be found using svcs -L:

root@logarchiver0 # svcs -L svc:/smartdc/application/hermes:default
/var/svc/log/smartdc-application-hermes:default.log

In these cases, once the server is up and running again, the logarchiver-agent service should be automatically installed.

Here are the log details for successful deployment of the logarchiver-agent to some servers:

[2019-06-27T07:47:15.859Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: starting periodic worker
[2019-06-27T07:47:15.974Z]  INFO: hermes/HttpServer/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: http server listening
    address: {
      "address": "10.99.99.71",
      "family": "IPv4",
      "port": 9999
    }
[2019-06-27T07:47:20.860Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: deploying actor (server=564d7de2-18d1-3c68-f893-44008b349c7d)
[2019-06-27T07:47:20.862Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: deploying actor (server=564d2b36-7e25-7d79-e3c2-7ceb79d8abd6)
[2019-06-27T07:47:20.863Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: deploying actor (server=564df69b-8c03-e569-0fe9-6a830b3f1084)
[2019-06-27T07:47:20.864Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: deploying actor (server=564d7861-5038-e426-d052-cbc059bf6083)
[2019-06-27T07:47:21.369Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: actor deployed (server=564df69b-8c03-e569-0fe9-6a830b3f1084)
[2019-06-27T07:47:21.408Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: actor deployed (server=564d7de2-18d1-3c68-f893-44008b349c7d)
[2019-06-27T07:47:21.408Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: actor deployed (server=564d7861-5038-e426-d052-cbc059bf6083)
[2019-06-27T07:47:21.556Z]  INFO: hermes/29734 on eb394c52-916e-4b95-aa53-3a7b6452824d: actor deployed (server=564d2b36-7e25-7d79-e3c2-7ceb79d8abd6)

DOCUMENTATION

APIs

Log Archiver Setup

Requirements

Service Creation

Agent Installation