Log Archiver Examples

The sdc zone and the logarchiver zone run different versions of the same hermes service, which will behave differently depending on which zone it is operating.

In general, sdc0 zone is used to archive Triton's Services generated logs, and logarchiver0 to archive user related log files. In the future, the hermes service will run just from the logarchiver0 zone.

Despite of the zone where the hermes service it's running, it can be configured using logsets in order to archive the selected log files.

A logset is a JSON object with some special properties, for example:

{
    "name": "firewall_logs",
    "search_dirs": [ "/var/log/firewall" ],
    "regex": "^/var/log/firewall/([0-9a-f]{8}\\-[0-9a-f]{4}\\-[0-9a-f]{4}\\-[0-9a-f]{4}\\-[0-9a-f]{12})/([0-9a-f]{8}\\-[0-9a-f]{4}\\-[0-9a-f]{4}\\-[0-9a-f]{4}\\-[0-9a-f]{12})/([0-9]+)-([0-9]+)-([0-9]+)T([0-9]+):([0-9]+):([0-9]+)\\.log.gz$",
    "manta_path": "/%U/reports/firewall-logs/#y/#m/#d/$2/#y-#m-#dT#H:#M:#S.log.gz",
    "customer_uuid": "$1",
    "date_string": {
        "y": "$3", "m": "$4", "d": "$5",
        "H": "$6", "M": "$7", "S": "$8"
    },
    "date_adjustment": "-1H",
    "debounce_time": 600,
    "retain_time": 0,
    "zones": [
        "global"
    ]
}

That logset tells the hermes service to search from the global zone into the directory /var/log/firewall for files matching the regular expression given by regex ($CUSTOMER_UUID/$VM_UUID/YY:MM:DDTHH:mm:SS.log.gz) and store into the given manta_path where %U is the user value set for customer_uuid property obtained from the regex, and the details about the date and time values used in the manta_path can be obtained from the date_string property which also uses the regex captured matches to set the values for the date-time parts.

This is the default logset used by logarchiver0 zone to capture user related log files for the machines the user has set log: true for firewall rules.

Example 1: Archive typical logs

A review of the logset file in use by the sdc0 zone includes the details about how to archive all the logs generated by a typical Triton setup. This file can be located at /opt/smartdc/hermes/etc/logsets.json. Here are two of the most interesting examples from that file that illustrate how to gather files for a given agent and for all the Triton zones for a known directory:

[
...
    {
        "name": "vm_agent_logs",
        "search_dirs": [ "/var/log/vm-agent" ],
        "regex": "^/var/log/vm-agent/vm-agent_([0-9a-zA-Z-]+)_([0-9]+)-([0-9]+)-([0-9]+)T([0-9]+):([0-9]+):([0-9]+)\\.log$",
        "manta_path": "/%u/stor/logs/%d/vm-agent/#y/#m/#d/#H/$1.log",
        "date_string": {
            "y": "$2", "m": "$3", "d": "$4",
            "H": "$5", "M": "$6", "S": "$7"
        },
        "date_adjustment": "-1H",
        "debounce_time": 600,
        "retain_time": 86400,
        "zones": [
            "global"
        ]
    },
        ...
    {
        "name": "sdc_zone_logs",
        "search_dirs": [ "/var/log/sdc/upload" ],
        "regex": "^/var/log/sdc/upload/([a-zA-Z0-9-]+)_([0-9a-f-]+)_([0-9]+)-([0-9]+)-([0-9]+)T([0-9]+):00:00\\.log$",
        "date_string": {
            "y": "$3", "m": "$4", "d": "$5",
            "H": "$6"
        },
        "date_adjustment": "-1H",
        "manta_path": "/%u/stor/logs/%d/$1/#y/#m/#d/#H/$2.log",
        "debounce_time": 600,
        "retain_time": 86400,
        "zones": [
            "adminui",
        "amon",
        "amonredis",
        "binder",
        "cloudapi",
        "cmon",
        "cnapi",
        "cns",
        "dhcpd",
        "docker",
        "fwapi",
        "imgapi",
        "mahi",
        "manatee",
        "manta",
        "moray",
        "napi",
        "papi",
        "portolan",
        "rabbitmq",
        "redis",
        "sapi",
        "sdc",
        "sdcsso",
        "ufds",
        "vmapi",
        "workflow"
            ]
    },
    ...
]

Example 2: Archive zone console logs

[
    {
        "name": "console_logs",
        "search_pattern": "/zones/*/logs",
        "search_dirs": [],
        "regex": "^/zones/([0-9a-zA-Z-]+)/logs/console\\.log\\.([0-9]+)$",
        "manta_path": "/%u/stor/console-logs/%d/$1/console.log",
        "debounce_time": 600,
        "retain_time": 21600,
        "date_string": {},
        "zones": [
            "global"
        ]
    }
]

Example 3: Archive docker logs

[
    {
        "name": "docker_logs",
        "search_pattern": "/zones/*/logs",
        "search_dirs": [],
        "regex": "^/zones/([0-9a-zA-Z-]+)/logs/([a-zA-Z0-9-]+)\\.log\\.([0-9]+)$",
        "manta_path": "/%u/stor/docker-logs/%d/docker-$1/$2/$1.log",
        "debounce_time": 600,
        "retain_time": 21600,
        "date_string": {},
        "zones": [
            "global"
        ]
    }
]

Example 4: Archive crash dumps

[
    {
        "name": "crash_dumps",
        "search_dirs": [ "/var/crash/volatile" ],
        "regex": "^/var/crash/volatile/(vmdump.(\\d+))$",
        "manta_path": "/%u/stor/dumps/%n/$1",
        "debounce_time": 600,
        "retain_time": 86400,
        "zones": [
            "global"
        ]
    }
]

Example 5: Archive core files

[
    {
        "name": "core_files",
        "search_dirs": [ "/zones/global/cores" ],
        "regex": "^/zones/global/cores/(core.([a-zA-Z0-9-]+).(\\d+))$",
        "manta_path": "/%u/stor/cores/%n/$1",
        "debounce_time": 600,
        "retain_time": 86400,
        "zones": [
            "global"
        ]
    }
]

Example 6: Remove docker stdio.log files without archiving them

Note the usage of attribute no_upload on this case.

[
    {
        "name": "docker_stdio_logs",
        "search_pattern": "/zones/*/logs",
        "search_dirs": [],
        "regex": "^/zones/([0-9a-zA-Z-]+)/logs/stdio\\.log\\.([0-9]+)$",
        "manta_path": "/%u/stor/logs/%d/docker-stdio/$1/stdio.log",
        "debounce_time": 600,
        "no_upload": true,
        "retain_time": 21600,
        "date_string": {},
        "zones": [
            "global"
        ]
    }
]