Skip to content

Europa Trouble Shooting QA notes

Jump to bottom Edit New page
kmurshed edited this page Jan 28, 2015 · 19 revisions
Steps to upgrade Control Center (Single Host)
Steps to upgrade Control Center (Multi-Host)
Steps to upgrade Zenoss:
  • Create a new text file by using vi tool e.g “vi upgrade.txt”
  • Add the following content to the text file:

# This script can be used to upgrade a Zenoss deployment to a new image # # Run via 'serviced script service Zenoss.resmgr upgrade.txt' and restart # the deployment

DESCRIPTION Zenoss RM 5.0.0_2071 upgrade VERSION resmgr-5.0.0_2071 REQUIRE_SVC SNAPSHOT

# Choose image to upgrade to SVC_USE zenoss/resmgr-unstable:5.0.0_2071

# Start all our dependent services SVC_START Zenoss.resmgr/MariaDB SVC_START Zenoss.resmgr/RabbitMQ SVC_START Zenoss.resmgr/zeneventserver SVC_START Zenoss.resmgr/zencatalogservice SVC_START Zenoss.resmgr/redis

# Wait for our services to start SVC_WAIT Zenoss.resmgr/MariaDB Zenoss.resmgr/RabbitMQ Zenoss.resmgr/zeneventserver Zenoss.resmgr/zencatalogservice Zenoss.resmgr/redis started 600

# Run the upgrade 'run' SVC_RUN Zenoss.resmgr/Zope upgrade

# Uncomment this to restart the entire application afterwards # SVC_RESTART Zenoss.resmgr

# Uncomment this if you have run the install_quilt script first, and are using # quilt (based in /opt/zenoss) to manage patches # SVC_RUN Zenoss.resmgr/Zope apply-custom-patches

After paste the content into the text file, change line number 6 “DESCRIPTION Zenoss RM 5.0.0_2071 upgrade” and replace the 2071 with the number of the image you want to use to upgrade Zenoss (those images can be found at: http://artifacts.zenoss.loc/europa/images/) and save the text file As root user execute: serviced script run upgrade.txt --service Zenoss.resmgr to start the upgrade process.

Important Notes:
  • It is not needed to stop Serviced or Zenoss to perform the upgrade.
  • if minions stop after the update just restart them
  • if after the upgrade of Zenoss one or more service health check is not blue, restart Zenoss,
  • if it doesn’t work stop zenoss, restart serviced then restart zenoss.
  • You can upgrade either Zenoss or Serviced or both depends of what you need.

Container:

How to attach to a container
  • serviced service attach service_name
  • Example: serviced service attach zenjobs
  • Example: serviced service attach dockerid
  • Example: serviced service attach zope/2 (this will connect to zope, 2nd instance)
How to find a file inside a container
  • Find the host where the container is running (Ex:look at CC-UI to see the service is running)
  • SSH to the host where the container is installed
  • then cd to the following directory: /var/lib/docker/aufs/mnt/ OR
  • Example: /var/lib/docker/vfs/dir/a274d524389af1d2d9e947b54f6148dcce3fc6d329328c88eb5c1dd21001e41a#/output.txt
  • You can copy this file to /tmp, if you want to export it to another host or attach to defect

How to pipe the contents of a command run inside a container to a file and then transfer out of container

Example:

serviced service attach zope/0

(enter password)

su - zenoss

zenmodeler run -d zdouglas-tb1 -v10 1> output.txt 2>&1

You can then exit the container and look for the file - NOTE: Be sure to look on the file system of the correct host if you are on multihost install.

find . -name "output.txt" (you can switch to /var and then search it's faster)

It will be in a folder like this:

/var/lib/docker/vfs/dir/a274d524389af1d2d9e947b54f6148dcce3fc6d329328c88eb5c1dd21001e41a#/output.txt

You may wish to copy the file from there to /tmp to make it easier to get out of Linux to Windows if needed.

Then use winscp or similar to copy the file from /tmp to windows if needed to attach to bug.

LOG FILES:

  • Serviced log is here: /var/log/upstart/serviced.log
  • Docker log is here: /var/log/upstart/docker.log
  • serviced service attach {service_name or docker id} --> many log files are under: /opt/zenoss/log
  • serviced log export --> this will zip all logs that you can attache to a ticket

System Usage: Memory, CPU, Disk Space, etc.

  • "top -c" or htop --> shows you CPU usage

  • "free -m" --> 2nd row (buffer/cache) shows you Memory used and Memory free

  • "vmstat 5 10" --> "si" shows you memory swap

  • "df -m" --> shows disk used and % free informations

  • Find memory used using opentsdb

  1. Login to opentsdb http://opentsdb.kmurshed-tb5:4242/
  2. add two metric: memory.actualused and swap.free
  3. make sure to monitor (see data) for past few days to see any memory leak/growth

Finding the memory usage and CPU of a service running in container (to verify graphs of a service in Europa)

Verify CPU

find PID of the service to test (top, L to locate process like metricshipper, note PID) pidstat -p {pid} 5 5

Note: You may need to average the result then multiply it by # of cores like %system value is .40 x 8 = 2.40 which should match the graph value assuming there are 8 cores on the system

VERIFY MEMORY

Find the container ID of the service you are looking for the memory of (look in control center, hosts, find the service, click on it, look at Docker Container ID

CD to dir with that container ID

/sys/fs/cgroup/memory/docker/aa143f2900a333d351968f5ad56c9245d14fde2f91eb92c7689da2ef02240176#

Cat file:

cat memory.stat

Look at value for, or grep, rss

This number should match with graph value at the same point in time

All services not starting?

Stop all services and start them one by one in the following order (if services does not stop, ssh to CC install box and stop serviced):

  • Zookeeper (wait long enough to make sure all instances are running)
  • HMaster
  • RegionServer
  • wait long enough until you can log into hbase URL

After all these are up, start the following:

  • opentsdb (reader and writer), wait until you can log into opentsdb URL
  • MySQL
  • RabbitMQ
  • CentralQuery
  • Zope
  • MetricConsumer
  • MetricShipper
  • Zauth
  • Rest

Getting hmtl code trying to login to Zenoss (RM) ?

  • RabbitMQ is probably down, restart it, look at the log to make sure it is up and login to zenoss again

Zenoss not showing any graph data ?

  • Wait for some time to see if graphs are rendering
  • CentralQuery - make sure service is running, look at log for any Error/Failure
  • MetricShipper - make sure service running, look at log for any Error/Failure
  • go to opentsdb URL, look to see if data is shown there (go to zenoss graph and 'inspect' to find the name of the parameter/metric to compare with that on opentsdb URL)

Finding RM Graph id to compare with Opentsdb

  • Go to RM graph (say: for rhel54 Linux using ssh) → right click the 1st graph→ Inspect Element
  • A pop up window opens (Chrome) → Click on Network → Wait till query/ appears on LHS
  • Click on the 1st query link (this corresponds to the first graph) → Click on Header (NOTE: if you are looking to compare 3rd graph on page, you have to find the 3rd “query” also look at the metrics name to see if it kind of matches, it would not be exact match but should be close)
  • Click/expand on metrics, under Request Payload section (this shows you 3 metrics test-rhel54.zenoss.loc/uptime_laLoadInt15, test-rhel54.zenoss.loc/uptime_laLoadInt5, test-rhel54.zenoss.loc/uptime_laLoadInt1)
  • Click on on the metric that you are interested say: first one i.e. 0 (zero)
  • This expands and shows you name of the metric (“test-rhel54.zenoss.loc/uptime_laLoadInt15”)
  • Expand tag shows you tags value pair (key: “Device/test-rhel54.zenoss.loc”)
  • Open vhost for opentsdb from CC UI
  • → Select time range where you want to see the graph
  • → Put metric name from above: test-rhel54.zenoss.loc/uptime_laLoadInt15
  • → Enter Tags from above: in 1st box --> key and second box --> Device/test-rhel54.zenoss.loc
  • → Wait/Refresh (as needed) → compare this data with that of RM Graph, these should match
  • .
  • Isvcs opentsdb can be found http://masterIP:4242

Forgot Zenoss (RM) password ?

  • attach to the zope container, run zenpass
  • Ignore if it says daemon manager not running, repeat if does not work first time OR
  • zpasswd.py -u admin -p meds22 inituser

Zookeeper Exhibitor

How to add SWAP to your AWS virtual machine

To add this extra space to your instance you type:

  • sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=15360
  • sudo /sbin/mkswap /var/swap.1
  • sudo /sbin/swapon /var/swap.1

If you need more than 16GB then change that to something higher.

To enable it by default after reboot, add this line to /etc/fstab:

  • /var/swap.1 swap swap defaults 0 0

How to pipe the contents of a command run inside a container to a file

Example:

serviced service attach zope/0

(enter password)

su - zenoss

zenmodeler run -d zdouglas-tb1 -v10 1> output.txt 2>&1

You can then exit the container and look for the file - NOTE: Be sure to look on the file system of the correct host if you are on multihost install.

find . -name "output.txt" (you can switch to /var and then search it's faster)

It will be in a folder like this:

/var/lib/docker/vfs/dir/a274d524389af1d2d9e947b54f6148dcce3fc6d329328c88eb5c1dd21001e41a#/output.txt

You may wish to copy the file from there to /tmp to make it easier to get out of Linux to Windows if needed.

Then use winscp or similar to copy the file from /tmp to windows if needed to attach to bug.

Installing Zenpack

  • Get the zenpack (egg) to a local directory (make sure this directory is writable)

  • cd to the directory where you copied the egg

  • Install the zenpack using (example): serviced service run zope zenpack install ZenPacks.zenoss.ControlCenter-blahversion.egg

  • ----- if this does not work try the following -------

  • serviced service attach zope

  • su - zenoss

  • cd to folder where you downloaded egg file

  • zenpack --install name_of_egg

  • restart zenoss resmgr (you may not need to restart all services but it is better)

  • Consult Install Guide for more detailed steps

Temporary way to install impact in Europa

Installing impact 5 on Europa

**MAKE SURE ZENOSS IMAGE HASN’T YET BEEN DEPLOYED

*Pull and tag impact image (The latest version is in https://registry.hub.docker.com/u/zenoss/impact-unstable/tags/manage/):

*$docker pull zenoss/impact-unstable:4.2.6.70.0_115 *$docker tag zenoss/impact-unstable:4.2.6.70.0_115 zenoss/impact-unstable:latest

*Deploy resmgr

*Start minimal services for ZenPack install: *$for svc in mysql rabbitmq redis zencatalogservice zeneventserver ; do serviced service start $svc ; done

*change directory to /opt/serviced

*Install ZenPacks: *$wget http://jenkins.zendev.org/view/Impact/job/ZenPacks.zenoss.ImpactServer-develop/lastSuccessfulBuild/artifact/dist/

*$wget http://jenkins.zendev.org/view/Impact/job/ZenPacks.zenoss.Impact-develop/lastSuccessfulBuild/artifact/

$serviced service run Zope zenpack install ZenPacks.zenoss.ImpactServer-.egg

*$serviced service start Impact

$serviced service run Zope zenpack install ZenPacks.zenoss.Impact-.egg

*Turn off the services that were previously turned on *Start all of the Europa services

Add a custom footer
Add a custom sidebar
Clone this wiki locally