Arborist-SNMP

code : http://code.martini.nu/Arborist-SNMP

mirrors : https://github.com/mahlonsmith/Arborist-SNMP : https://hg.sr.ht/~mahlon/Arborist-SNMP

Description

Arborist is a monitoring toolkit that follows the UNIX philosophy of small parts and loose coupling for stability, reliability, and customizability.

This adds various SNMP support to Arborist's monitoring, specifically for OIDS involving:

• Disk space capacity
• System CPU utilization
• Memory and swap usage
• Running process checks
• UPS battery checks

It tries to provide sane defaults, while allowing fine grained settings per resource node. Both Windows and UCD-SNMP systems are supported.

Prerequisites

• Ruby 2.4 or better

Installation

$ gem install arborist-snmp

Configuration

Global configuration overrides can be added to the Arborist config file, under the snmp key.

The defaults are as follows:

arborist:
  snmp:
    timeout: 2
    retries: 1
    community: public
    version: 2c
    port: 161
    batchsize: 25
    cpu:
      warn_at: 80
    disk:
      warn_at: 90
      include: 
      exclude:
      - "^/dev(/.+)?$"
      - "/dev$"
      - "^/net(/.+)?$"
      - "/proc$"
      - "^/run$"
      - "^/sys/"
      - "/sys$"
    processes:
      check: []
    memory:
      physical_warn_at: 
      swap_warn_at: 60
    ups:
      battery:
        capacity_warn_at: 60
        temperature_warn_at: 50

The warn_at keys imply usage capacity as a percentage. ie: "Warn me when a disk mount point is at 90 percent utilization."

Library Options

• timeout: How long to wait for an SNMP response, in seconds?
• retries: If an error occurs during SNMP communication, try again this many times before giving up.
• community: The SNMP community name for reading data.
• version: The SNMP protocol version. v1, v2c, and v3 are supported.
• port: The UDP port SNMP is listening on.
• batchsize: How many hosts to gather SNMP data on simultaneously.

Category Options and Behavior

CPU

• warn_at: Set the node to a warning state when utilization is at or over this percentage.

Utilization takes into account CPU core counts, and uses the 5 minute load average to calculate a percentage of current CPU use.

2 properties are set on the node. cpu contains the detected CPU count and current utilization. load contains the 1, 5, and 15 minute load averages of the machine.

Disk

• warn_at: Set the node to a warning state when disk capacity is at or over this amount. You can also set this to a Hash, keyed on mount name, if you want differing warning values per mount point. A mount point that is at 100% capacity will be explicity set to down, as the resource it represents has been exhausted.
• alert_readonly: Set the now to an error state if mounts are readonly. This may be helpful in iscsi-root environments. You can set this to a Hash, keyed on mount name, if you want differing behavior per mount point.
• include: String or Array of Strings. If present, only matching mount points are considered while performing checks. These are treated as regular expressions.
• exclude: String or Array of Strings. If present, matching mount point are removed from evaluation. These are treated as regular expressions.

A single property "mounts" is set on the node, which is a hash keyed by mountpoint, with current capacity values.

Memory

• physical_warn_at: Set the node to a warning state when RAM utilization is at or over this percentage.
• swap_warn_at: Set the node to a warning state when swap utilization is at or over this percentage.

Warnings are only set for swap by default, since that is usually a better indication of an impending problem.

2 properties are set on the node, "memory" and "swap". Each is a Hash that contains current usage and remaining available.

Processes

• check: String or Array of Strings. A list of processes to check if running. These are treated as regular expressions, and include process arguments.

If any process in the list is not found in the process table, the resource is set to a down state.

A single property is set on the node, a "counts" key that contains the current number of running processes.

UPS

Battery

• capacity_warn_at: Set the node to a warning state when the battery capacity is at or below this percentage.
• temperature_warn_at: Set the node to a warning state when the battery temperature is at or above this temperature (in Celcius).

This check warns when the UPS is on battery, and reports the UPS' own estimated time remaining.

A warning state will also be triggered based on battery status - low, depleted, or unknown.

Examples

In the simplest form, using default behaviors and settings, here's an example Monitor configuration:

require 'arborist/snmp'

Arborist::Monitor 'cpu load check', :cpu do
	every 1.minute
	match type: 'resource', category: 'cpu'
	exec( Arborist::Monitor::SNMP::CPU )
end

Arborist::Monitor 'partition capacity', :disk do
	every 1.minute
	match type: 'resource', category: 'disk'
	exec( Arborist::Monitor::SNMP::Disk )
end

Arborist::Monitor 'process checks', :proc do
	every 1.minute
	match type: 'resource', category: 'process'
	exec( Arborist::Monitor::SNMP::Process )
end

Arborist::Monitor 'memory', :memory do
	every 1.minute
	match type: 'resource', category: 'memory'
	exec( Arborist::Monitor::SNMP::Memory )
end

Additionally, if you'd like these SNMP monitors to rely on the SNMP service itself, you can add a UDP check for that.

Arborist::Monitor 'udp service checks', :udp do
	every 30.seconds
	match type: 'service', protocol: 'udp'
	exec( Arborist::Monitor::Socket::UDP )
end

And a default node declaration:

Arborist::Host 'example' do
	description 'An example host'
	address 'demo.example.com'

	resource 'cpu'
	resource 'memory'
	resource 'disk'
end

All configuration can be overridden from the defaults using the config pragma, per node. Here's a more elaborate example that performs the following:

• All SNMP monitored resources are quieted if the SNMP service itself is unavailable.
• Only monitor specific disk partitions, warning at different capacities .
• Ensure the 'important' processing is running with the '--production' flag.
• Warns at 95% memory utilization OR 10% swap.
• Ensure '/' is not a mounted as a read-only filesystem.

Arborist::Host 'example' do
	description 'An example host'
	address 'demo.example.com'

	service 'snmp', protocol: 'udp'

	resource 'cpu', description: 'machine cpu load' do
		depends_on 'example-snmp'
	end

	resource 'memory', description: 'machine ram and swap' do
		depends_on 'example-snmp'
		config physical_warn_at: 95, swap_warn_at: 10
	end

	resource 'disk', description: 'partition capacity' do
		depends_on 'example-snmp'
		config \
			include: [
				'^/tmp',
				'^/var'
			],
			warn_at: {
				'/tmp' => 50,
				'/var' => 80
			},
			alert_readonly: {
				'/' => true
			}
	end

	resource 'process' do
		depends_on 'example-snmp'
		config check: 'important --production'
	end
end

License

Copyright (c) 2016-2020 Michael Granger and Mahlon E. Smith All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

• Neither the name of the author/s, nor the names of the project's contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.