Telegraf as part of the TIG Stack

The TIG Stack is an open-source and modern approach to infrastructure and application monitoring that uses the lightweight Telegraf agent for data collection and submission to an InfluxDB time-series optimised database, and the Grafana dashboard tool for visualisation and alerts.

TIG Stack is a great way for organisations of all sizes to perform incredibly powerful monitoring of on-premises and cloud infrastructure at minimal cost. The Telegraf agent is available for Windows and is in the repositories for most Linux distributions, meaning almost every server can run it.

However, I've encountered use cases where Telegraf is not available for a Linux-based server that needs to be monitored. At work, these include software appliances such as HP OneView, Dell OpenManage, and in my home lab I have JeOS appliances such as Libre-Elec and Home Assistant OS on Raspberry Pis.

In these instances, a simple shell script can be using in place of the Telegraf agent to perform monitoring by following the principles below. As an example, here is how I monitor Libre-Elec, although the approach is suitable for any server:

Create and schedule a monitoring script

Using the command line (or maybe WinSCP if you prefer a GUI):

  • Login to your Libre-Elec server using SSH
  • cd ~ to ensure you are in your home directory, should be /storage
  • Make a scripts folder by typing mkdir scripts
  • Enter your scripts folder cd scripts
  • Create a new monitoring script using your favourite text editor, e.g.  nano ./monitoring.sh
  • Amend the proforma script below, replacing <yourInfluxDbServer> with the hostname or IP of your InfluxDB server, <yourMonitoringDb> with the name of your monitoring database and <yourMonitoredHost> with the hostname of the computer you are monitoring.
#!/bin/sh

#Monitor boot and root SD Card capacities
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "mmcblk0p1_pcUsed,host=<yourMonitoredHost> value=`df | grep mmc.*p1 | awk '{print $5}' | sed 's/%//'`"
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "mmcblk0p2_pcUsed,host=<yourMonitoredHost> value=`df | grep mmc.*p2 | awk '{print $5}' | sed 's/%//'`"
#Monitor USB Pendrive capacities
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "sda_pcUsed,host=<yourMonitoredHost> value=`df | grep sda | awk '{print $5}' | sed 's/%//'`"
#Monitor CPU Temperature
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "cpu_temp,host=<yourMonitoredHost> value=`/opt/vc/bin/vcgencmd measure_temp |  sed 's/temp=//' | sed 's/.C//'`"
#Monitor memory capacity
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "mem_free,host=<yourMonitoredHost> value=`free -h --mega | grep Mem. | awk '{print $4}' | sed 's/M//'`"
#Monitor CPU Capacities
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "cpu_user,host=<yourMonitoredHost> value=`mpstat | grep all | awk '{print $3}'`"
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "cpu_sys,host=<yourMonitoredHost> value=`mpstat | grep all | awk '{print $5}'`"
#Monitor uptime
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "uptime_seconds,host=<yourMonitoredHost> value=`awk '{print $1}' /proc/uptime`"
#Monitor power supply stability
curl -i -XPOST 'http://<yourInfluxDbServer>:8086/write?db=<yourMonitoringDb>' --data-binary "voltage_status,host=<yourMonitoredHost> value=` /opt/vc/bin/vcgencmd get_throttled | sed 's/throttled=0x//'`"
  • Make the script executable by running chmod +x ./monitoring.sh
  • Test the script by calling it explicitly, e.g. ./monitoring.sh; ensure the InfluxDB returns status 200 for each CURL command.
  • Schedule the script by editing crontab crontab -e
  • Schedule the monitoring script to run on a repeating schedule, e.g. every five minutes */5 * * * *     /storage/scripts/monitoring.sh

Future improvements

I really should parameterise the <yourInfluxDbServer> and <yourMonitoringDb> tokens, and automatically populate the <yourMonitoredHost> token by calling hostname.
This could be useful for monitoring VMWare ESXi without vSphere, only VMWare do not provide CURL as a command line tool. What a shame.