monitoring_utils

xsrv.monitoring_utils

This role will install and configure various monitoring and audit utilities:

• lynis security auditing tool
• htop system monitor/process manager
• nethogs network bandwidth monitor
• ncdu disk usage viewer
• lnav log viewer
• (optional) duc disk usage analyzer
• (optional) bonnie++ disk benchmarking tool

Requirements/dependencies/example playbook

See meta/main.yml

- hosts: my.CHANGEME.org
  roles:
    - nodiscc.xsrv.common # (optional) basic setup, hardening, firewall
    - nodiscc.xsrv.monitoring_utils
    # - nodiscc.xsrv.monitoring # (optional) full monitoring suite including monitoring_utils

See defaults/main.yml for all configuration variables

Usage

• Show htop process manager: ssh -t user@my.CHANGEME.org sudo htop
• Analyze disk usage by directory: ssh -t user@my.CHANGEME.org sudo ncdu /
• Show network bandwidth usage by process: ssh -t user@my.CHANGEME.org sudo nethogs
• Show network connections: ssh -t user@my.CHANGEME.org sudo watch -n 2 ss -laptu
• Visualize disk usage by directory: TAGS=utils-duc xsrv deploy default my.CHANGEME.org and run duc gui --database=data/duc-my.CHANGEME.org.db / on the controller (requires duc)
• Run disk read/write speed benchmarks: TAGS=utils-bonnie xsrv deploy default my.CHANGEME.org and open data/bonnie++-my.CHANGEME.org.html on the controller. You can compare reports from multiple hosts using cat data/bonnie++*.csv | bon_csv2html > bonnie++.html.
• Use lnav to navigate/search/filter aggregated system logs:

# using https://xsrv.readthedocs.io/en/latest/
xsrv logs [project] [host]
# using ssh
ssh -t user@my.CHANGEME.org sudo lnav /var/log/syslog

Useful lnav commands:

• :filter-in <expression> only display messages matching filter expression
• :set-min-log-level debug|info|warning|error only display messages above a defined log level.
• :<TAB><TAB> display internal command list
• Ctrl+R clear all filters/reset session
• ? lnav help
• q exit lnav

Read system logs as a non-root/sudoer user: add your user to the adm group. Example using the ../common role:

linux_users:
   - name: "{{ ansible_user }}"
     groups: adm
     append: yes
     comment: "ansible user/allowed to read system logs"

Disk benchmarking: bonnie++ is configured to run the benchmark with test files of 1024MB, and will allocate a set amount of memory of 512MB.

Tags

lynis - setup lynis security audit tool
monitoring_utils - setup command-line/additional monitoring utilities
utils-duc - (manual) run duc disk usage analyzer and download the report on the controller
utils-bonnie - (manual) run duc disk benchmarking tool and download the report on the controller

License

GNU GPLv3

References

• https://stdout.root.sx/links/?searchtags=monitoring
• https://stdout.root.sx/links/?searchtags=security