====== Icinga2 ====== Go to the [[https://icinga.cm.in.tum.de | Icinga Chair Website.]] ===== Configuration ===== All configuration files are under the directory ''/etc/icinga2/conf.d''\\ The configuration is done in three parts. - Setup the ''host'' information: vms.conf / servers.conf - Manage the services and how they are checked (check attributes): services.conf - Low level execution of scripts and commands: commands.conf Normally it is enough to fill out the host information. The services are automatically applied to any hosts that fit the scheme. The Nagios Monitoring scripts are located under\\ ''/usr/lib/nagios/plugins/''. ==== Configure a new machine ==== These things get checked: ping, ssh, cpu-load, disk-usage\\ **Client** - Install the nagios plugin package sudo apt-get install nagios-plugins-basic - Create a new user icinga2


sudo adduser --disabled-password --gecos "" icinga2

- Setup SSH Public Key authentication


sudo mkdir /home/icinga2/.ssh
sudo vim /home/icinga2/.ssh/authorized_keys

- Add following part to the authorized_keys file:


command="/usr/lib/nagios/plugins/check_disk `echo $SSH_ORIGINAL_COMMAND | grep '^-w [0-9,%]* -c [0-9,%]*'` -A -I /sys/kernel/debug/* -I /var/lib/docker/* -I /run/docker/*",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9jOn1brQW5uItt8By/JWaoSnrOU9uruHAIXpIBX1rb6xunezkr4vO7AkppnK9AkbeYqGn2YvKZMFim8UW+wgKoO6jFIbp3ha5uyoL8R2lzRYmnKOkCBRZIJfUjlKUcPXTNgGmYl6QUx20DM2vfcoJ070bK8LoEO06nEddmYQ6RQcm4jO4uUOxdRgQ2WEF+F9fg5R3Qyff2bvLTH0QUqWcgnZu1SPhF+Xzm4PTHpx+d3RR35V5uArhjGmGV6ZPaCpKKPUUtFNx4rTwsQm9z0zUK0r7TyXGQf3+s5ybdMKws8dyGhTPETYxqD1wrcJzw+bhBdvnnmw+OfxERCeO0U2RoO4aJS+CKLJF6l1urKY051TDf3QfLvJNPdy2kE7D4pNSPlj8TJ0FyFlZ8trRYKTdpJekQRYCSwlQmpp2Q/VkgtcdomIb4vPHbxWOaM1crMCAxzSAptVqVjND1ouZA8jtcT9xP8YeG1B2lUQPqwRHtn/nM9rfwrvgQ02WLgYua9QnLmoK6xExIrt5wI44ez3wTBnQSWnMrYrcz1P8YlK4s8rqHeFICulIOsW6h36aP2ijuBkjr/p9sXn/Ki4OgJ5E/lNT90P+AaaYXQwpJocubutPiXgb2YIjnOciltk6BhsboP2xVbW7VheSRsJjkCgiRvzXELG66vAecYjRCkoIPQ== icinga2@icinga
command="/usr/lib/nagios/plugins/check_load `echo $SSH_ORIGINAL_COMMAND | grep '^-w [0-9,]* -c [0-9,]*'`",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDLHbR0NfP5Bas921FK+qcOCLIH716F8dZ6gsO85Ot2NYxKtDIuNEER9h6p3IYGcLLGlWNELyvc4B6O7ityKRA5PLPvxD5XbVp/CjKsdBMMfDJHVD9g9ZW8fCIDDabxpxHGa227fTJMcgXL7VM7qkIx8Sn619bpSGk5MQzv942YqOJ+JrT/11OgERRmqBaXCWAtnX7cyaPUgUDgeR0hmGddyUhOlJvgkX34V8WGNwttJ4vRu/oGqZ0TKPQfKHLsnBzRAwZpEILxGKool0e1VnWuAvkCIK/wVcOxB1y6FKYyYxLarMshwEEzaiE8eaWbwwaGTj4ejftA/rBPZAGUydshWMtuBzIDDXXD2t+Xt9iDS2HCDTTbucn36JbecZ+pdtKoTW9Wo3PbzqPqmJtNEQizIIf+1dboM1MP1eumIGF8XOZpKlTA8+Ola9ItQKMoTMVVCVlTPnk6x8ug2ocX9ykC+12xLpZoaWuzQMbtVyl4C7h1KB1svt5DCCJZDuM1FVwC3wPyUqypZ50dRekwvi+lHRBvSaj0xl/MfQufJiD3wjsj0Y5fSbIKexntvJ/VDq78s2beWFbro8+RN7e09T3Qe2tr3jEQbIBkyfEKdXtLbxVpncT+A6u/QWktO0ZN5g7yFuZbfg+iF21vwgp+2R9IHFkeEqVdYGI5L+0f/qjabQ== icinga2@icinga


command="/usr/lib/nagios/plugins/check_disk `echo $SSH_ORIGINAL_COMMAND | grep '^-w [0-9,%]* -c [0-9,%]*'` -A -I /sys/kernel/debug/* -I /var/lib/docker/* -I /run/docker/*",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDYSC1aCXTje+8/4fkq5njBoaQwy0WHtspBtYu86KF7yxdat5kTbKnbjwAGDkM+YfMq5u+tf6ehPIffJVI/6pnhBfZ6aTKNqj+VAgD9DBHdiEi+H6p8rmdf480zSAPRTLE6YG/Ca58hotBc23v/99Ud/7ofdWpUCWCLaV5P8SAyLAHfYftyt457BfPCnPkkyYutYKwz02nsNaFtiK3XxGIvvBy3epUuCR+LuZIdg6CcJIZpwu1fGYWGDIbKf/VzUqULoYy/0Zlo9JZlgOdtlNo+6Y404e0qRbVxwrUFUNnEmjCsrVICEmWQoKBQe9T6ShicCURYCcSKEk15ZRYveny62jr2ybC+Tm7Qx94dXb4F8SNLbrsQwjF8mwmyGtYMw1kb1kiuY7+o7zZHJJ6U3gbGf2e21PpP7vrKcquUGO82Gn4LhDbW2BOwuyXUW+2Y8FOR20eFox6er3CeO5rhu9V68aaIchkK5h4+izuHiLKFkEsvSnbuMhCvsWgirjNrSGLZD1W4SbUbFAYq1CP8sBA/dESjstfki4/I2ZZyZ0I2o0erYKiMFpKGEiAD1gc266RVNcD41/rfsb0eFlFI+OZctMINZmr2ZeH4oELinovNF/KWoPvj00wavdxEw/IeeyLz4Jjk7GuYmgVBIeQwUdW7Uo770GkR1eIIh8eZZ1juNQ== icinga2@icinga
command="/usr/lib/nagios/plugins/check_load `echo $SSH_ORIGINAL_COMMAND | grep '^-w [0-9,]* -c [0-9,]*'`",no-port-forwarding,no-x11-forwarding,no-agent-forwarding ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdwLRhCohYPnOjyq+QQQ35hTXZJ9/va82IPcQLdE671x+gaGZadOgceO/ymq5ge7dI9ZiBBHeMbI6gqZHTHJCJmQxVcqJ32pRvDzcgi8RnsfLUyQXZulRhFb+Chcaw8lV2R0fJMN5Q4+UhjVNrumx6JDooiinYWEgMBSyo8HIFBs/SWE39Z/FlqDUgyDa9q02TiNkvFBYgLeoDAIyxcdVC6LX8FIjCvlOZLozhQYhmQLzJYhJmpQvIJ4q7SP+zOc08nHHhXqC8h4RvjSImGrAagQd9N5esykaIHbSQglv0bV544Lxms+Fe2MaNKfw3KixNhRvlYmLYSA32tffUWwOLfrKSj96fEeDm9pfL9dLHxJ+1CsvJb1kSwfZoJ71bZg3nu3nQPi4LBpBb8aLiD+e+4mCQGpYZmc9eQ+7iJj5jOucB54HVBHea5DpPZ6foSdJEilq8ziv69uqhWEwQxy8m7MJgKcV/7bNuqID5sbGh893UhJ4mZjRLGsXlEChhA+om+TnBogCYN8UByg/ONpoxYSEqzeJzopMMxa/b4WkJroMr80S+aJKQJnmymP1LMaWkJiLHEsoulmuEGVENM34uFLAb733ChNpVibp0WSCZHoue/CAT9/1wlM3m93z+85yFLHtt1M2XCjky+yYEEQ03mW5zLIzgN6zzOOdPvSFZ5w== icinga2@icinga

- Change permission of ssh files


sudo chown -R icinga2:icinga2 /home/icinga2/.ssh/

**Host** - Add a new section to the vms.conf/servers.conf


object Host "new_machine - purpose" {
  address = ""
  vars.os = "Linux"
  check_command = "hostalive"
  vars.type = "disk-vm, load-vm / disk-server, load-server"
  #optional values for disk check
  vars.ssh_command_disk = " -w 50% -c 10%"
  #optional values for load check
  vars.ssh_command_load = " -w 5,4,3 -c 10,6,4"
}

Load = /\\ -w WLOAD1,WLOAD5,WLOAD15 -c CLOAD1,CLOAD5,CLOAD15\\ WLOAD1: threshold one minute WLOAD5 threshold last five minutes WLOAD15: threshold last 15 minutes\\ If one process uses CPU 100% system has load = 1, two processes use CPU 100% system has load = 2, etc. -w percent free disk space -c percent free disk space\\ e.g.: -w 10% -c 5% = warn when only 10% disk space left, critical when only 5% left\\ ===== Installation ===== This section lists the commands to install icinga2. Execute all commands with a prepended sudo or in a administrator shell.


apt-get install software-properties-common
add-apt-repository ppa:formorer/icinga
apt-get update
apt-get install icinga2

Now the commands for icinga2 web:


apt-get install mysql-server mysql-client
#set mysql root password
apt-get install icinga2-ido-mysql
#choose no
mysql -u root -p
mysql> create database icinga; grant all on icinga.* to 'icinga'@'localhost' identified by '';
mysql -u icinga -p icinga < /usr/share/icinga2-ido-mysql/schema/mysql.sql
icinga2 feature enable ido-mysql
icinga2 feature enable command
vim /etc/icinga2/features-enabled/ido-mysql.conf
#fill out the password, user, database fields
service icinga2 restart
------------------------
wget -O - http://packages.icinga.org/icinga.key | apt-key add -
add-apt-repository 'deb http://packages.icinga.org/ubuntu icinga-trusty main'
apt-get update
apt-get install icingaweb2
#some steps because of php7.0
a2dismod mpm_event
a2enmod mpm_prefork
a2enmod php7.0
service apache2 restart
icingacli setup token create
#show token in case you forgot
icingacli setup token show

visit this webpage [[http://icinga.cm.in.tum.de/icingaweb2/setup]] In the settings we change the php timezone to a fixed values

sudo vim /etc/php/7.0/apache2/php.ini
#change this line
date.timezone = "Europe/Berlin"
#install some additional php packages to get graphs working
apt-get install php7.0-intl
apt-get install php7.0-gd
apt-get install php7.0-xml

In the further configuration choose LDAP as the authentification backend:


LDAP RESOURCE
Host: ldap://ldapswitch.informatik.tu-muenchen.de
Port: 389
Root DN: ou=Personen,ou=IN,o=TUM,c=DE
AUTHENTICATION BACKEND
Backend Type: LDAP
Ldap User Object Class: rbgAccount
LDAP User Name Attribute: uid
USER GROUP BACKEND
LDAP Group Object Class: posixGroup
ldap Group Filter : |(gidNumber=5440)(gidNumber=13457)
LDAP Group Name Attribute: cn
LDAP Group Member Attribute: memberUid
LDAP Base DN: ou=Gruppen,ou=IN,ou=TUM,c=DE

Now you need to configure the database access for icingaweb2. Just put in all the information and passwords you got from the step above while installing the icinga2 main component. If you got some weird permission errors resolve them:


chown -R www-data:icingaweb2 /etc/icingaweb2/modules

---- Set up ip routes so the il11 network (edison network, wifi) is reachable. This is used to monitor devices in the il11 network (e.g. edison sensor devices). On the il11 gateway (vmott3) the firewall has to be set up accordingly to allow commands from the icinga host through the gateway to the network devices. sudo ip route add 172.24.21.192/27 Add this code to the interfaces file to persist virtual machine reboots.

 sudo vim /etc/network/interfaces
iface ens160 inet dhcp
    up ip route add 172.24.21.192/27 via 131.159.24.141 || true

===== Checks Setup ===== ==== Ping ==== - Address parameter and hostalive check_command has to be set in servers.conf / vms.conf


object Host "machine" {
  address = "131.159.24.1"
  check_command = "hostalive"
}

==== SSH ==== - Address parameter and vars.os has to be set in servers.conf / vms.conf


object Host "machine" {
  address = "131.159.24.1"
  vars.os = "Linux"
}

==== Disk Usage ==== - vars.type needs a disk-vm or disk-server:


object Host "machine" {
  address = "131.159.24.1"
  vars.type = "disk-vm/disk-server"
  #optional line with disk parameters
  vars.ssh_command_disk = " -w 10% -c 5%"
}


apply Service "ssh_disk_server" {
  import "generic-service"
  check_command = "ssh_disk_server"
  display_name = "disk"

  assign where match("*disk-server*", host.vars.type)
}
apply Service "ssh_disk_vm" {
  import "generic-service"
  check_command = "ssh_disk"
  display_name = "disk"

  assign where match("*disk-vm*", host.vars.type)
}


object CheckCommand "ssh_disk_server" {
        import "plugin-check-command"

        command = [ PluginDir + "/check_by_ssh" ]

        arguments = {
                "-H" = "$ssh_hostname$"
                "-C" = "$ssh_command_disk$"
                "-l" = "$ssh_remote_user$"
                "-i" = "$ssh_identity$"
                }
        vars.ssh_hostname = "$address$"
        vars.ssh_remote_user = "icinga2"
        vars.ssh_identity = "/home/icinga2/.ssh/id_rsa_server_disk"
        vars.ssh_command_disk = " -w 10% -c 5%"
}
object CheckCommand "ssh_disk" {
        import "plugin-check-command"

        command = [ PluginDir + "/check_by_ssh" ]

        arguments = {
                "-H" = "$ssh_hostname$"
                "-C" = "$ssh_command_disk$"
                "-l" = "$ssh_remote_user$"
                "-i" = "$ssh_identity$"
                }
        vars.ssh_hostname = "$address$"
        vars.ssh_remote_user = "icinga2"
        vars.ssh_identity = "/home/icinga2/.ssh/id_rsa_vm_disk"
        vars.ssh_command_disk = " -w 10% -c 5%"
}

In systems more recent than 15.04 there is a permission big when checking the filesystem. \\ ''DISK CRITICAL - /run/lxcfs/controllers is not accessible: Permission denied''\\ There is a workaround:


sudo chown root:root /usr/lib/nagios/plugins/check_disk
sudo chmod u+s /usr/lib/nagios/plugins/check_disk
sudo chmod o+x /usr/lib/nagios/plugins/check_disk

==== CPU-Load ==== - host needs in vars.type a "load-vm/load-server":


object Host "machine" {
  address = "131.159.24.1"
  vars.type = "load-vm/load-server"
  #optional line with load parameters
  vars.ssh_command_load = " -w 5,4,3 -c 10,6,4"
}


apply Service "ssh_load_server" {
  import "generic-service"
  check_command = "ssh_load_server"
  display_name = "load"
  
  assign where match("*load-server*", host.vars.type)
} 
apply Service "ssh_load" {
  import "generic-service"
  check_command = "ssh_load"
  display_name = "load"

  assign where match("*load-vm*", host.vars.type)
}


object CheckCommand "ssh_load_server" {
        import "plugin-check-command"

        command = [ PluginDir + "/check_by_ssh" ]

        arguments = {
                "-H" = "$ssh_hostname$"
                "-C" = "$ssh_command_load$"
                "-l" = "$ssh_remote_user$"
                "-i" = "$ssh_identity$"
                }
        vars.ssh_hostname = "$address$"
        vars.ssh_remote_user = "icinga2"
        vars.ssh_identity = "/home/icinga2/.ssh/id_rsa_server_load"
        vars.ssh_command_load = " -w 5,4,3 -c 10,6,4"
}
object CheckCommand "ssh_load" {
        import "plugin-check-command"

        command = [ PluginDir + "/check_by_ssh" ]

        arguments = {
                "-H" = "$ssh_hostname$"
                "-C" = "$ssh_command_load$"
                "-l" = "$ssh_remote_user$"
                "-i" = "$ssh_identity$"
                }
        vars.ssh_hostname = "$address$"
        vars.ssh_remote_user = "icinga2"
        vars.ssh_identity = "/home/icinga2/.ssh/id_rsa_vm_load"
        vars.ssh_command_load = " -w 5,4,3 -c 10,6,4"
}

==== Website ==== Check HTTP Website for String and certificates. First install nagios-plugins for check_http: \\ sudo apt install nagios-plugin **Http Check**: http_host (req), http_url (default: /), http_ssl (default: true), http_string (req, string to check for) \\ **Http Certificate Check**: http_host (req), http_url (defaul: /), http_cert_exp (default: 14,7, first warn value then critical value days till expiration)\\


object Host "one02 - edge,phi - webserver - ding" {
  address = "131.159.24.86"
  vars.os = "Linux"
  check_command = "hostalive"
  vars.http["Edge Computing Workshop 2017"] = {
        http_host = "edge17.cm.in.tum.de"
        http_string = "Mobile Networking, Analytics and Edge Computing"
        http_ssl = "true"
  }
  vars.http["EdgeSys Workshop 2018"] = {
        http_host = "edgesys18.cm.in.tum.de"
        http_string = "The 1st International Workshop on Edge Systems"
        http_ssl = "true"
  }

  vars.http_cert["Edge Computing Workshop 2017 - Certificate"] = {
        http_cert_exp = "14,7"
        http_host = "edge17.cm.in.tum.de"
  }
  vars.http_cert["EdgeSys Workshop 2018 - Certificate"] = {
        http_cert_exp = "14,7"
        http_host = "edgesys18.cm.in.tum.de"
  }
}


object ServiceGroup "http" {
  display_name = "HTTP Checks"

  assign where match("http*", service.check_command)
}

object ServiceGroup "http_certificates" {
  display_name = "HTTPS Certificate Checks"

  assign where match("http_cert*", service.check_command)
}


object CheckCommand "http_check" {
        import "plugin-check-command"

        command = [ PluginDir + "/check_http" ]

        arguments = {
                "-H" = "$http_host$"
                "-u" = "$http_url$"
                "-s" = "$http_string$"
                "-S" = {
                  description = "Enable SSL/TLS"
                  set_if = "$http_ssl$"
                }
                "-f" = {
                 value= "$http_redirect$"
                 description="Output when redirected (default: warning)"
                }
        }
        vars.http_url = "/"
        vars.http_redirect = "warning"
        vars.http_ssl = "true"
}
object CheckCommand "http_certificate" {
        import "plugin-check-command"

        command = [ PluginDir + "/check_http" ]

        arguments = {
                "-H" = "$http_host$"
                "-C" = "$http_cert_exp$"
                "-u" = "$http_url$"
        }
        vars.http_cert_exp = "14,7"
        vars.http_url = "/"
}


apply Service for (http_host => config in host.vars.http) {
  import "generic-service"

  check_command = "http_check"
  vars += config
}

apply Service for (http_host => config in host.vars.http_cert) {
  import "generic-service"

  check_command = "http_certificate"
  vars += config
}

===== Other ===== **Icinga Config Changes**\\ - Check if configs are correct


sudo service icinga2 checkconfig

- Reload icinga config files


sudo service icinga2 reload

** Icinga directories **\\ Main directory with config files: ''/etc/icinga2/conf.d/''