ZenPacks

OpenStack (Provider View) ZenPack

This ZenPack allows for monitoring of OpenStack from a service provider perspective. This means that in addition to the user-oriented components supported in the regular OpenStack ZenPack (instances, flavors, images), the underlying OpenStack servers and software are monitored.

Prerequisites

This Zenoss-specific plugin must be installed on all your Ceilometer nodes as described. If not installed and configured properly, Instances and vNICs graphs will be blank, and Zenoss will not detect changes (such as new instances or instance state changes) until a full model is performed.

Supported OpenStack Releases

Post-Installation Notes

Once the zenpack is installed, provide SSH credentials to the linux devices in your OpenStack environment before adding any devices.

Configure the zCommandUsername/zCommandPassword/zKeyPath properties on the /Devices/Server/SSH/Linux/NovaHost device class.

If your OpenStack nodes are already managed under Zenoss, move them into /Devices/Server/SSH/Linux/NovaHost

After upgrading the ZenPack, if running Zenoss 5.2.x, you may get a flare message with the error “TypeError: addOpenStack() got an unexpected keyword argument ‘ceilometer_url’”. This is due to a caching issue in Zenoss. To resolve the issue, restart the container for the Zenoss service (which runs the zproxy server that handles web requests). You do not need to restart the child services, only the parent. After restarting once, the issue should not recur.

Installed Items

Configuration Properties

zOpenStackAuthUrl: The URL of the Identity endpoint.

zOpenStackExtraHosts: The list of extra hosts that will be added to the system once OpenStack Infrastructure device is modeled.

zOpenStackExtraApiEndpoints: A list of URLs to monitor for openstack APIs. Format is <service type>:<full url> for each.

zOpenStackHostDeviceClass: Used as a default device class for defined hosts in zOpenStackExtraHosts and zOpenStackNovaApiHosts properties. Default is /Server/SSH/Linux/NovaHost.

zOpenStackNovaApiHosts: The list of hosts upon which nova-api runs. This is required when the IP address in the nova API url does not match any known host.

zOpenStackCinderApiHosts: The list of hosts upon which cinder-api runs. This is required when the IP address in the cinder API url does not match any known host.

zOpenStackHostMapToId: A list of <name>=<id>, used to force a host referred to by openstack with the given name to be represented in Zenoss as a host component with the given ID. (this is not commonly used)

zOpenStackHostMapSame: A list of <name1>=<name2>, used to inform the modeler that the same host may be referred to with an alternate name by some part of openstack. (this is not commonly used)

PerfCeilometerAPIDataSource: Used to capture datapoints from OpenStack Ceilometer.

QueueSizeDataSource: Checks the number of unprocessed messages in Ceilometer AMQP queues.

Monitoring Templates

/OpenStack/Infrastructure/

Endpoint

Instance

vNIC

Event Classes and Mappings

/OpenStack

OpenStack Events Default

/OpenStack/Cinder

/OpenStack/Cinder/Snapshot

Cinder Snapshot default mapping

/OpenStack/Cinder/Volume

cinder.volume default mapping

/OpenStack/compute

/OpenStack/compute/instance

compute.instance default mapping

compute.instance.create.error

compute.instance.exists

compute.instance.exists.verified.old

/OpenStack/dhcp_agent

dhcp_agent default mapping

/OpenStack/firewall

firewall default mapping

/OpenStack/firewall_policy

firewall_policy default mapping

/OpenStack/firewall_rule

firewall_rule default mapping

/OpenStack/floatingip

floatingip default mapping

/OpenStack/network

network default mapping

/OpenStack/port

port default mapping

/OpenStack/router

router default mapping

/OpenStack/security_group

security_group default mapping

/OpenStack/security_group_rule

security_group_rule default mapping

/OpenStack/subnet

subnet default mapping

/Status/Heartbeat/

openStackCeilometerHeartbeat

/Status

openStackCinderServiceStatus

openStackIniFileAccess

openStackIniFileOptionParsing

openStackNeutronAgentStatus

openStackNovaServiceStatus

openStackApiEndpointStatus

Processes

/OpenStack

/OpenStack/ceilometer-agent-central

/OpenStack/ceilometer-agent-compute

/OpenStack/ceilometer-agent-notification

/OpenStack/ceilometer-alarm-evaluator

/OpenStack/ceilometer-alarm-notifier

/OpenStack/ceilometer-api

/OpenStack/ceilometer-collector

/OpenStack/ceilometer-polling

/OpenStack/cinder-api

/OpenStack/cinder-backup

/OpenStack/cinder-scheduler

/OpenStack/cinder-volume

/OpenStack/glance-api

/OpenStack/glance-registry

/OpenStack/gnocchi-metricd

/OpenStack/gnocchi-statsd

/OpenStack/keystone-admin

/OpenStack/keystone-all

/OpenStack/keystone-main

/OpenStack/neutron-dhcp-agent

/OpenStack/neutron-l3-agent

/OpenStack/neutron-lbaas-agent

/OpenStack/neutron-metadata-agent

/OpenStack/neutron-metering-agent

/OpenStack/neutron-openvswitch-agent

/OpenStack/neutron-server

/OpenStack/nova-api

/OpenStack/nova-cert

/OpenStack/nova-compute

/OpenStack/nova-conductor

/OpenStack/nova-consoleauth

/OpenStack/nova-network

/OpenStack/nova-scheduler

/OpenStack/rabbitmq-server

Basic Usage

The OpenStackInfrastructrue ZenPack models vNICs associated with OpenStack Instances. In order to correctly model these vNICs, you must first fully model the OpenStack environment and then configure and model the /Server/SSH/Linux/NovaHost devices. See the section below for details on configuration specifics.

Device Setup via UI

Once the OpenStack ZenPack is installed and you can begin monitoring by going to the infrastructure screen and clicking the normal button for adding devices. You’ll find a new option labeled, “Add OpenStack Endpoint (Infrastructure).”

Choose that option and you’ll be presented with a dialog asking for the following inputs.

Device To Create - non-empty, non-ip, non-dns, unique name to use for this device in Zenoss. ‘’See note below’’.

Auth URL - A keystone URL. For Keystone’s v3 API, it should look like http://\<hostname\>:5000/v3/. For Keystone’s v2 API, it should look like http://\<hostname\>:5000/v2.0/. To have the ZenPack choose the newest supported API version, leave the path off, like http://\<hostname\>:5000/.

Region Name - choose the correct region from the drop-down. You may only choose one, so each region you wish to manage must be registered as a separate endpoint in Zenoss.

Once you click Add, Zenoss will contact the OpenStack API and discover servers, images and flavors. Once it is complete you’ll find a new device in the OpenStack device class with the same name as the hostname or IP you entered into the dialog. Click into this new device to see everything that was discovered.

{{note|’‘’Device Name’’‘}} The’‘’Device name’’’ should be a non-empty, non-hostname, non-IP, since that name will be used when the host is registered as a Linux device. The name should be unique within the Zenoss environment. This is especially important if you are adding another device that share the same IP address or hostname that already exist on another device. Not doing this may result in devices with the same name conflicting with each other. (e.g. attempting to model device would show modeling results that belong to another device OR device would show relations that do not belong to that device)

Note: All events processed through Ceilometer are automatically exposed via the Zenoss Event Console, and all metrics collected by Ceilometer may be collected and graphed in Zenoss through the use of custom monitoring templates.

Daemons

Type

Name

Modeler

zenmodeler

Performance Collector

zencommand, zenpython

Ceilometer Enablement

Although you may add an OpenStack device to Zenoss, as shown above, event and performance data will not be collected until the following steps are performed.

Zenoss Configuration Steps - First-Time Installation

The first time you install this zenpack, you must run openstack_amqp_config to create the RabbitMQ exchanges that are used to integrate with ceilometer.

To run this script, log into the master server (Zenoss 4.x) or Zope container (zenoss 5.x) and run it as follows:

If not already set, populate the zOpenStackAMQPUsername and zOpenStackAMQPPassword zProperties. (Generating a random password)

Create the required AMQP exchanges in RabbitMQ if they are missing

Register the user from zOpenStackAMQPUsername in rabbitmq and update its password to match zOpenStackAMQPPassword.

Display the configuration parameters that you will need to add to the ceilometer.conf file on your OpenStack servers (see below.)

You may safely re-run openstack_amqp_config at any time to display the configuration parameters, or to update the username/password after you have changed zOpenStackAMQPUsername and zOpenStackAMQPPassword

Zenoss Configuration Steps - ZenPack Upgrades

If this is the first time you are upgrading from a version of the zenpack prior to 2.4.0, you should first set zOpenStackAMQPUsername and zOpenStackAMQPPassword to match the values of amqp_userid and amqp_password from ceilometer.conf on your openstack systems.

If you do not do this, if you run openstack_amqp_config in the future, it will generate a new password and reconfigure rabbitmq to use that password instead of the one you have been using, which would interrupt monitoring.

For upgrade from version 2.4.0 or higher, there are no special steps required, no changes required on the OpenStack side, and no need to run openstack_amqp_config.

NOTE: After upgrading the ZenPack, if running Zenoss 5.2.x, you may get a flare message with the error “TypeError: addOpenStack() got an unexpected keyword argument ‘ceilometer_url’”. This is a known issue, due to a caching issue in Zenoss. To resolve the issue, restart the container for the Zenoss service (which runs the zproxy server that handles web requests). You do not need to restart the child services, only the parent. After restarting once, the issue should not recur.

Zenoss 5.x - RabbitMQ-Ceilometer

Version 2.4.0 of this zenpack introduces a new service, RabbitMQ-Ceilometer. This is a dedicated instance of RabbitMQ on each collector which is used solely for integration with ceilometer, rather than using the standard RabbitMQ service that is used by Zenoss itself. This better distributes any load as well as providing better support for distributed collector scenarios where the target OpenStack environment might not have network access to the central Zenoss servers.

This ZenPack still supports the previous configuration, where messages were sent from ceilometer to the main RabbitMQ service in zenoss, so if you were previously using it that way successfully, there is no need to reconfigure your ceilometer.conf to point it at the new location.

For new openstack installs, we recommend using the RabbitMQ-Ceilometer endpoint, which is what will be reported by openstack_amqp_config.

When the RabbitMQ-Ceilometer service is added to Control Center, two IP Assignments will be added as well. One on port 55672 for the RabbitMQ node, and one on port 45672 for the management console. Upon creation, they may be missing their IP addresses, in which case, they will default to automatic IP assignment. In order to function properly, these services must have static IPs assigned. If they have no IP address (only a port number), you must add the IP address by following these steps:

In Control Center, open the service page for your Zenoss instance.

Click “Assign” for the node IP assignment (port 55672).

Select the appropriate IP from the “IP” select box on the “Assign IP” modal dialog.

Click “Assign IP”.

Repeat for the management console IP Assignment (45672).

NOTE: Control Center services will not be able to restart successfully until the IP addresses are added.

OpenStack Ceilometer Configuration Steps

Zenoss relies upon a Ceilometer dispatcher plugin to ship raw event and metering data from Ceilometer to Zenoss for storage in the Zenoss event and performance databases. This integration is done by publishing messages to Zenoss’s RabbitMQ server.

This dispatcher should be installed on all nodes running any ceilometer, but particularly those running ceilometer-collector or ceilometer-agent-notification.

Ceilometer_zenoss must be installed on all ceilometer nodes in the openstack environment. To install the latest released version from RPM:

Then, ensure that the configuration options output by the openstack_amqp_config script previously are added to /etc/ceilometer/ceilometer.conf file on all openstack nodes.

Restart all Ceilometer services on all hosts after making these changes.

Optional Steps

VM State Changes

By default, instance state changes will be captured by Zenoss when certain events occur, for example, when an instance is shut down, the state change to SHUTDOWN will be reflected in Zenoss.

However, certain state changes that don’t correspond to another defined event may not be picked up until the next time Zenoss models the environment.

If you would like to reduce the likelihood of this occurring, you can configure OpenStack Nova to send an event (through ceilometer) to Zenoss whenever any VM state change occurs by adding the following to /etc/nova/nova.conf on all Nova hosts:

[DEFAULT]
notify_on_state_change=vm_and_task_state

For Liberty:

[oslo_messaging_notifications]
notification_driver = messagingv2

For Mitaka:

[oslo_messaging_notifications]
driver = messagingv2

Save /etc/nova/nova.conf and restart nova services.

Note that notify_on_state_change will cause increased event load, both on OpenStack and Zenoss, and additional processing within the event transforms in Zenoss to keep the model consistent. Since most instance changes will still be caught without this option enabled, it is recommended to leave notify_on_state_change disabled if your OpenStack environment is very large.

Network Events

The OpenStack Infrastructure ZP pulls neutron events for Networks and Routers. However, if Neutron is not configured properly to send those events, they cannot be retrieved. If you are missing events, checks your OpenStack environment’s /etc/neutron/neutron.conf. It should have the following:

Increasing Polling Interval

Zenoss will process performance datapoints from Ceilometer every 10 minutes, since by default, Ceilometer will only produce one datapoint every 10 minutes. This can be adjusted by modifying the “interval: 600” line in your pipeline.yaml file (typically /etc/ceilometer/pipeline.yaml).

Troubleshooting

Host Identification

The openstack APIs do not contain an authoritative list of hosts with unique IDs. Instead, various APIs show hosts by name or IP. There zenpack does its best to identify IPs and names that refer to the same host, and represent them as a single host component. In some cases, though, it can’t tell, and the same host may be modeled twice, or with an incorrect name.

Two zProperties are provided to override the default behavior of the zenpack when this happens.

zOpenStackHostMapSame

Specifies that two names refer to the same host. It is a list of entries of the form: <name1>=<name2> For example,

my.example.com=myothername.example.com
my.example.com=10.1.1.1

This means that any time the host “my.example.com”, “myothername.example.com”, or “10.1.1.1” is encountered, they will be considered to be the same host, rather than separate ones.

zOpenStackHostMapToId

It is also possible to specify not only that the devices are the same, but that they should be identified with one specific identifier (otherwise, one may be chosen at random). In this case, a list of entries of the form <name>=<id> may be provided in the zOpenStackHostMapToId zProperty. For example, myothername.example.com=my.example.com 10.1.1.1=my.example.com This would cause “my.example.com”, “myothername.example.com”, or “10.1.1.1” to all be definitely identified as “my.example.com”, without the ambiguity that could exist if zOpenStackHostMapSame were used.

zOpenStackHostLocalDomain

In some environments (in particular, the Red Hat OpenStack Platform), hosts are assigned names that end in ‘.localdomain’. This would cause problems for zenoss, because it is not possible to create a device in zenoss with such a name, as they all resolve to 127.0.0.1, rather than their actual IP.

The default value of zOpenStackHostLocalDomain is a blank string, meaning that the ‘.localdomain’ suffix will be stripped from host names, and devices will be created in zenoss with those shortened names. If those names do not resolve in DNS, they will be created without IPs, and will not be modeled. You would need to manually set their management IPs so that they can be modeled.

Alternatively, if you already have these hostnames in dns, but just with a different domain name than “.localdomain”, you may specify this domain name here, and it will be substituted for localdomain, and the devices will model automatically, based on the IPs returned from DNS.

Modeling Containerized Environments

If the target openstack environment runs processes inside of docker containers, it is necessary to configure several zProperties before modeling will succeed.

You can now navigate back to the ‘’OpenStackInfrastructure ZenPack’’ folder in the repository to see the following resources added by the bundle.

Domains

OpenStackInfrastructure Domain

Ad Hoc Views

OpenStack Instance List

The OpenStackInfrastructure Domain can be used to create ad hoc views using the following steps.

Choose Ad Hoc View from the Create menu.

Click Domains at the top of the data chooser dialog.

Expand Public then OpenStackInfrastructure ZenPack.

Choose the OpenStackInfrastructure Domain domain

Service Impact and Root Cause Analysis

When combined with the Zenoss Service Dynamics product, this ZenPack adds built-in service impact and root cause analysis capabilities for OpenStack infrastructure and instances. The service impact relationships shown in the diagram and described below are automatically added. These will be included in any services that contain one or more of the explicitly mentioned components.

Recommended Impact Setup

Since most components will be related to Tenants and Region we recommend:

Examples

Impact (Instance)

Impact (Network)

Impact (Region)

Impact (Tenant)

Integration with other ZenPacks

In some cases, the underlying network or storage technology is monitored by a different zenpack. The OpenStackInfrastructure zenpack is able to integrate with the following zenpacks to provide component-level linkage and impact model integration:

Known Issues

ZEN-14585: The same endpoint can not be monitored both as user and an infrastructure endpoints.

Workaround: If you have been previously monitoring the endpoint as a User endpoint, delete the device before you re-add it as an Infrastructure endpoint.

ZEN-17905: Nova APIs component: Grey icons for Enabled and State after model/monitor.

OpenStack nova service API does not provide information about Nova-API, so its status is, in fact, unknown.

ZPS-1762: When using OpenvSwitch integration, the Linux devices must be added to the system first (normally through automatic discovery by the OpenStackInfrastructure ZenPack) before the corresponding OpenvSwitch devices are registered. This is because the two devices use the same management IP address, and a special exclusion is in place for OpenvSwitch devices, allowing them to be added after the linux device, but not the other way around.

ZPS-1956: After upgrading the ZenPack, if running Zenoss 5.2.x, you may get a flare message with the error “TypeError: addOpenStack() got an unexpected keyword argument ‘ceilometer_url’”. This is due to a caching issue in Zenoss. To resolve the issue, restart the container for the Zenoss service (which runs the zproxy server that handles web requests). You do not need to restart the child services, only the parent. After restarting once, the issue should not recur.

ZPS-2004: When adding an OSI device, if the same host is already added as a generic device (such as /SSH/Linux), the host device’s device class will be changed, and an error generated, preventing modeling. As a workaround, remove the Linux device before adding the OSI device.

Changes

2.4.2

Avoid nameconfict for proxy devices and be more flexible in linking to existing devices when appropriate. (ZPS-3991)