OpenStack – Service Engineering (ICCLab & SPLab)https://blog.zhaw.ch/icclab
A Blog of the ZHAW Zurich University of Applied Sciences
Tue, 12 Mar 2019 15:42:03 +0000 en-US
hourly
1 https://wordpress.org/?v=5.1.1SC2 2018 – The 8th IEEE International Symposium on Cloud and Services Computinghttps://blog.zhaw.ch/icclab/sc2-2018-the-8th-ieee-international-symposium-on-cloud-and-services-computing/
https://blog.zhaw.ch/icclab/sc2-2018-the-8th-ieee-international-symposium-on-cloud-and-services-computing/#commentsFri, 23 Nov 2018 14:01:03 +0000https://blog.zhaw.ch/icclab/?p=12222Continue reading →]]>The 8th IEEE International Symposium on Cloud and Services Computing (IEEE SC2) 2018, took place in Paris, France, from the 19th to 22nd of November. The conference was co-located with two more events, namely the 5th International Conference on Internet of Vehicles (IOV) 2018 and the 11th IEEE International Conference on Service-Oriented Computing and Applications (IEEE SOCA) 2018.

We from the ICCLab participated for first three days of the SC2 conference as its focus on Cloud related topics meets our expertise and research interests for research and development activities. The main themes in focus were Cloud Platforms and Services, Networking and Services, and Cloud and SOA Services.

As an important venue for researchers and industry practitioners, SC2 offered the opportunity to exchange information about recent advancements for IT-driven cloud computing technologies and services. The conference hosted a good number of participants for a familiar context were interacting with peers was easy, not in the last place over the coffee and lunch breaks. In total 68 oral presentations were planned in 15 sessions. Additionally 10 posters were presented in a poster session and 3 keynotes were organized. The conference was organized over three days, with the first day being dedicated to tutorials and the next days with parallel sessions for each of the co-located conferences.

Besides attending the event itself, the main motivation to visit the SC2 conference was to present our paper entitled “Hera Object Storage: A seamless, Automated Multi-Tiering Solution on Top of Openstack Swift”. In the presented paper we highlighted some of our recent results from research in the field of Cloud storage. In particular, the focus of the contribution is in the fast-growing field of unstructured data storage in distributed cloud. We proposed an object storage solution built on top of OpenStack Swift. This solution is able to apply a multi-tiering storage to unstructured data in a seamless and automatic manner. The object storage decisions are taken based on the data temperature, in terms of current access rate.

The first day of the conference, was the day of my arrival. The first tutorial I could attend gave insights in the NVIDIA company and their activities in the automotive industry. Various interesting results were presented, supported by real-world test videos. We could see how NVIDIA as a market leader supports manufacturers in building self-driving cars. We could appreciate how a full range of real-world conditions influencing the traffic conditions could be handled. The amount of work behind these results was probably not completely clear to many of us, but the needed hardware and software infrastructure was clearly huge!

The second very interesting talk was showing an interesting business model presented by Qarnot computing, France. The model they presented promoted a solution where computing and heating are delivered from a cloud infrastructure. The solution is based on a geo-distributed cloud platform with server nodes named digital heaters. Each heater embeds processors or GPU cards and is connected to the heat diffusion system. With this solution, homes, offices and other buildings can be heated through the distributed data center which is able to balance the requests in computation and heating.

The last tutorial of the day proposed some basics of machine learning for unsupervised algorithms. A review on the applications and the challenges faced when dealing with data sets was also given.

The second day started with the official opening of the conferences with the presentation of the program. This was followed by a keynote on scheduling methods for elastic services as for a project driven by the AlterWay company in France. The rest of the day we had two conference sessions and a further Keynote speech where Cybersecurity, with its links to geopolitical issues, was in focus.

In the first SC2 session I attended we could follow presentations about the following topics: a comparison between unikernels and containers, user plane management for 5G networks, a cost analysis of virtual machine live migration, and two papers on automated tiered storage solutions, one of which I presented myself. The second session was dedicated to work in progress papers, covering topics like contextual information searching for encrypted data in cloud storage services, smart contracts with on and off-blockchain components, cloud native 5G virtual network functions.

The evening we could enjoy a banquet on the Seine river with all the conference attendees. The cruise on the Seine brought us close to the main sightseeing attractions of beautiful Paris. A fish-based dinner was served completing a perfect environment to exchange experiences with other conference participants.

Day 3 started with an enlightening keynote speech of Prof. Cesare Pautasso from the University of Lugano, Switzerland, which described the recent trend in terms of software development. This is dictated by the current scenario where end-users have multiple devices to access their data and contents and managing their personal information. To best manage such a complex multi-device user environment Liquid software is needed, whereby software can seamlessly flow and adapt to the different devices.

After the last session with some interesting papers presenting among others solutions for multi-objective scheduling in cloud computing, confidentiality and privacy issues in the Cloud, it was time to head back home. Our participation to the SC2 conference was definitely positive and we will surely consider next year’s conference edition as possible venue to share our new research experience.

]]>https://blog.zhaw.ch/icclab/sc2-2018-the-8th-ieee-international-symposium-on-cloud-and-services-computing/feed/1Experience using Kolla Ansible to upgrade Openstack from Ocata to Queenshttps://blog.zhaw.ch/icclab/experience-using-kolla-ansible-to-upgrade-openstack-from-ocata-to-queens/
https://blog.zhaw.ch/icclab/experience-using-kolla-ansible-to-upgrade-openstack-from-ocata-to-queens/#respondFri, 27 Jul 2018 13:54:26 +0000https://blog.zhaw.ch/icclab/?p=12122Continue reading →]]>We made a decision to use Kolla-Ansible for Openstack management approximately a year ago and we’ve just gone through the process of upgrading from Ocata to Pike to Queens. Here we provide a few notes on the experience.

By way of some context: our system is a moderate sized system with 3 storage nodes, 7 compute nodes and 3 controllers configured in HA. Our systems were running CentOS 7.5 with a 17.05.0-ce docker engine and we were using the centos-binary Kolla containers. Being an academic institution, usage of our system peaks during term time – performing the upgrade during the summer meant that system utilization was modest. As we are lucky enough to have tolerant users, we were not excessively concerned with ensuring minimal system downtime.

We had done some homework on some test systems in different configurations and had obtained some confidence with the Kolla-Ansible Ocata-Pike-Queens upgrade – we even managed to ‘upgrade’ from a set of centos containers to ubuntu containers without problem. We had also done an upgrade on a smaller, newer system which is in use and it went smoothly. However, we still had a little apprehension when performing the upgrade on the larger system.

In general, we found Kolla Ansible good and we were able to perform the upgrade without too much difficulty. However, it is not an entirely hands-off operation and it did require some intervention for which good knowledge of both Openstack and Kolla was necessary.

Our workflow was straightforward, comprising of the following three stages

generate the three configuration files passwords.yml, globals.yml and multinode.ha,

pull down all containers to the nodes using kolla-ansible pull

perform the upgrade using kolla-ansible upgrade.

We generated the globals.yml and passwords.yml config files by copying the empty config files from the appropriate kolla-ansible git branch to our /etc/kolla directory, comparing them with the files used in the previous deploy and copying changes from the previous versions into the new config file. We used the approach described here to generate the correct passwords.yml file.

It can take a bit of time, but it’s sensible as it does not have any impact on the operational system and reduces the amount of downtime when upgrading.

We were then ready to perform the deployment. Rather than run the system through the entire upgrade process, we chose a more conservative approach in which we upgraded a single service at a time: this was to maintain a little more control over the process and to enable us to check that each service was operating correctly after upgrade. We performed this using commands such as:

We stepped through the services in the same order as listed in the main Kolla-Ansible playbook, deploying the services one by one.

The two services that we were most concerned about were those pertaining to data storage, naturally: mariadb and ceph. We were quite confident that the other processes should not cause significant problems as they do not retain much important state.

Before we started…

We had some initial problems with docker python libraries installed on all of our nodes. The variant of the docker python library available via standard CentOS repos is too old. We had to resort to pip to install a new docker python library which worked with newer versions of Kolla-Ansible.

Ocata-Pike Upgrade

Deploying all the services for the Ocata-Pike upgrade was straightforward: we just ran through each of the services in turn and there were no specific issues. When performing some final testing, however, the compute nodes were unable to schedule new VMs as neutron was unable to attach a VIF to the OVS bridge. We had seen this issue before and we knew that putting the compute nodes through a boot cycle solves it – not a very clean approach, but it worked.

Pike-Queens Upgrade

The Pike-Queens upgrade was more complex and we encountered issues that we had not specifically seen documented anywhere. The issues were the following:

the mariadb upgrade failed – when the slave instances were restarted, they did not join the mariadb cluster and we ended up with a cluster with 0 nodes in the ‘JOINED’ state. The master node also ended up in an inoperational state.

We solved this using the well documented approach to bootstrapping a mariadb cluster – we have our own variant of it for the kolla mariadb containers, which is essentially a replica of the mariadb_recovery functionality provided by kolla

This did involve a syncing process of replicating all data from the bootstrap node on each of the slave nodes; in our case, this took 10 minutes

when the mariadb database sync’d and reached quorum, we noticed many errors associated with record field types in the logs – for this upgrade, it was necessary to perform a mysql_upgrade, which we had not seen documented anywhere

the ceph upgrade process was remarkably painless, especially given that this involved a transition from Ceph Jewel to Ceph Luminous. We did have the following small issues to deal with

We had to modify the configuration of the ceph cluster using ceph osd require-osd-release luminous

We had one small issue that the cluster was in the HEALTH_WARN status as one application did not have an appropriate tag – this was easily fixed using ceph osd pool application enable {pool-name} {application-name}

for reasons that are not clear to us, Luminous considered the status of the cluster to be somewhat suboptimal and moved over 50% of the objects in the cluster; Jewel had given no indication that a large amount of the cluster data needed to be moved

Upgrading the object store rendered it unusable: in this upgrade, the user which authenticates against keystone with privilege to manage user data for the object store changed from admin to ceph_rgw. However, this user was not added to the keystone and all requests to the object store failed. Adding this user to the keystone and giving this user appropriate access to the service project fixed the issue.

This was due to a change that was introduced in the Ocata release after we had performed our deployment and it only became visible to use after we performed the upgrade.

Apart from those issues, everything worked fine; we did note that the nova database upgrade/migration in the Pike-Queens cycle did take quite a long time (about 10 minutes) for our small cluster – for a very large configuration, it may be necessary to monitor this more closely.

Final remarks…

The Kolla-Ansible upgrade process worked well for our modest deployment and we are happy to recommend it as an Openstack management tool for environments of such scale with quite standard configurations, although even with an advanced tool such as Kolla-Ansible, it is essential to have a good understanding of both Openstack and Kolla before depending on it in a production system.

]]>https://blog.zhaw.ch/icclab/experience-using-kolla-ansible-to-upgrade-openstack-from-ocata-to-queens/feed/0Setting up container based Openstack with OVN networkinghttps://blog.zhaw.ch/icclab/setting-up-container-based-openstack-with-ovn-networking/
https://blog.zhaw.ch/icclab/setting-up-container-based-openstack-with-ovn-networking/#respondMon, 09 Jul 2018 09:03:11 +0000https://blog.zhaw.ch/icclab/?p=12101Continue reading →]]>OVN is a relatively new networking technology which provides a powerful and flexible software implementation of standard networking functionalities such as switches, routers, firewalls, etc. Importantly, OVN is distributed in the sense that the aforementioned network entities can be realized over a distributed set of compute/networking resources. OVN is tightly coupled with OVS, essentially being a layer of abstraction which sits above a set of OVS switches and realizes the above networking components across these switches in a distributed manner.

A number of cloud computing platforms and more general compute resource management frameworks are working on OVN support, including oVirt, Openstack, Kubernetes and Openshift – progress on this front is quite advanced. Interestingly and importantly, one dimension of the OVN vision is that it can act as a common networking substrate which could facilitate integration of more than one of the above systems, although the realization of that vision remains future work.

In the context of our work on developing an edge computing testbed, we set up a modest Openstack cluster, to emulate functionality deployed within an Enterprise Data Centre with OVN providing network capabilities to the cluster. This blog post provides a brief overview of the system architecture and notes some issues we had getting it up and running.

As our system is not a production system, providing High Availability (HA) support was not one of the requirements; consequently, it was not necessary to consider HA OVN mode. As such, it was natural to host the OVN control services, including the Northbound and Southbound DBs and the Northbound daemon (ovn-northd) on the Openstack controller node. As this is the node through which external traffic goes, we also needed to run an external facing OVS on this node which required its own OVN controller and local OVS database. Further, as this OVS chassis is intended for external traffic, it needed to be configured with ‘enable-chassis-as-gw‘.

We configured our system to use DHCP provided by OVN; consequently the neutron DHCP agent was no longer necessary, we removed this process from our controller node. Similarly, L3 routing was done within OVN meaning that the neutron L3 agent was no longer necessary. Openstack metadata support is implemented differently when OVN is used: instead of having a single metadata process running on a controller serving all metadata requests, the metadata service is deployed on each node and the OVS switch on each node routes requests to 169.254.169.254 to the local metadata agent; this then queries the nova metadata service to obtain the metadata for the specific VM.

The services deployed on the controller and compute nodes are shown in Figure 1 below.

Figure 1: Neutron containers with and without OVN

We used Kolla to deploy the system. Kolla does not currently have full support for OVN; however specific Kolla containers for OVN have been created (e.g. kolla/ubuntu-binary-ovn-controller:queens, kolla/ubuntu-binary-neutron-server-ovn:queens). Hence, we used an approach which augments the standard Kolla-ansible deployment with manual configuration of the extra containers necessary to get the system running on OVN.

As always, many smaller issues were encountered while getting the system working – we will not detail all these issues here, but rather focus on the more substantive issues. We divide these into three specific categories: OVN parameters which need to be configured, configuration specifics for the Kolla OVN containers and finally a point which arose due to assumptions made within Kolla that do not necessarily hold for OVN.

To enable OVN, it was necessary to modify the configuration of the OVS switches operating on all the nodes; the existing OVS containers and OVSDB could be used for this – the OVS version shipped with Kolla/Queens is v2.9.0 – but it was necessary to modify some settings. First, it was necessary to configure system-ids for all of the OVS chassis’ – we chose to select fixed UUIDs a priori and use these for each deployment such that we had a more systematic process for setting up the system but it’s possible to use a randomly generated UUID.

Having changed the OVS configuration on all the nodes, it was then necessary to get the services operational on the nodes. There are two specific aspects to this: modifying the service configuration files as necessary and starting the new services in the correct way.

Not many changes to the service configurations were required. The primary changes related to ensuring the the OVN mechanism driver was used and letting neutron know how to communicate with OVN. We also used the geneve tunnelling protocol in our deployment and this required the following configuration settings:

For the metadata agent container (running on the compute nodes) it was necessary to configure it to point at the nova metadata service with the appropriate shared key as well as how to communicate with OVS running on each of the compute nodes

For the OVN specific containers – ovn-northd, ovn-sb and ovn-nb databases, it was necessary to ensure that they had the correct configuration at startup; specifically, that they knew how to communicate with the relevant dbs. Hence, start commands such as

The last issue which we had to overcome was that Kolla and neutron OVN had slightly different views regarding the naming of the external bridges. Kolla-ansible configured a connection between the br-ex and br-int OVS bridges on the controller node with port names phy-br-ex and int-br-ex respectively. OVN also created ports with the same purpose but with different names patch-provnet-<UUID>-to-br-int and patch-br-int-to-provonet-<UUID>; as these ports had the same purpose, our somewhat hacky solution was to manually remove the the ports created in the first instance by Kolla-ansible.

Having overcome all these steps, it was possible to launch a VM which had external network connectivity and to which a floating IP address could be assigned.

Clearly, this approach is not realistic for supporting a production environment, but it’s an appropriate level of hackery for a testbed.

Other noteworthy issues which arose during this work include the following:

Standard docker apparmor configuration in ubuntu is such that mount cannot be run inside containers, even if they have the appropriate privileges. This has to be disabled or else it is necessary to ensure that the containers do not use the default docker apparmor profile.

A specific issue with mounts inside a container which resulted in the mount table filling up with 65536 mounts and rendering the host quite unusable (thanks to Stefan for providing a bit more detail on this) – the workaround was to ensure that /run/netns was bind mounted into the container.

As we used geneve encapsulation, geneve kernel modules had to be loaded

Full datapath NAT support is only available for linux kernel 4.6 and up. We had to upgrade the 4.4 kernel which came with our standard ubuntu 16.04 environment.

This is certainly not a complete guide to how to get Openstack up and running with OVN, but may be useful to some folks who are toying with this. In future, we’re going to experiment with extending OVN to an edge networking context and will provide more details as this work evolves.

]]>https://blog.zhaw.ch/icclab/setting-up-container-based-openstack-with-ovn-networking/feed/0How to Write a Cinder Driverhttps://blog.zhaw.ch/icclab/how-to-write-a-cinder-driver/
https://blog.zhaw.ch/icclab/how-to-write-a-cinder-driver/#respondMon, 12 Jun 2017 12:01:34 +0000https://blog.zhaw.ch/icclab/?p=11360Continue reading →]]>After too many hours of trial and error and searching for the right solution on how to properly write and integrate your own backend in cinder, here are all the steps and instructions necessary. So if you are looking for a guide on how to integrate your own cinder driver, look no further.

Why do we need a Cinder Driver, and why are we even using Cinder? We created Hera, a distributed storage system, based on ZFS, used in SESAME, a 5G project, in which we are project partners. To achieve an integration of Hera into SESAME, which uses OpenStack, we had to create a Cinder driver.

First of all, we have the Hera storage system with a RESTful API, all the logic and functionality is already available. We position the driver as a proxy between Cinder and Hera. To implement the driver methods, one does not have to look very far, there is a page on the OpenStack Cinder docs that explain which methods need to be implemented and what they do. For a basic Cinder Driver skeleton check out this repository: Cinder Driver Example.

We have decided for a normal volume driver, but you may also decide for another driver that you want to write, then you need to inherit from another base driver, e.g.: write your Driver for a san volumes (SanDriver) or for iSCSI volumes (ISCSIDriver). Also we have always looked at other drivers (mainly the LVM driver) for some guidance during the implementation.

These methods are necessary for a complete driver, while implementing it we wanted to try single methods after implementing them. Once the mandatory methods were implemented, and we attempted to execute the driver’s code, nothing was happening! We quickly realised, that the get_volume_stats method returns crucial information of the storage system to the Cinder scheduler. The scheduler will not know anything of the driver if values are not returned, so to quickly test we had this dict hardcoded and the scheduler stopped complaining.

In order to provider parameters to your driver, you can also add them in the following way, as part of the driver implementation. Here, we add a REST endpoint as a configuration option to the volume_opts part.

All of the options that are defined, can be overwritten by putting them inside the /etc/cinder/cinder.conf file under the configuration of our own driver.

In order to understand what values Cinder will give to a driver the volume parameter can be used. When you get to implement the functionality of the driver, you will want to know what is passed to the driver by cinder, especially the volume dict parameter is of great interest, and it will have these values:

size
host
user_id
project_id
status
display_name
display_description
attach_status
availability_zone
// and if any of the following are set
migration_status
consistencygroup_id
group_id
volume_type_id
replication_extended_status
replication_driver_data
previous_status

To test your methods quickly and easily, it is very important that the driver is in the correct directory, in which all the Cinder drivers are installed, otherwise Cinder will, naturally, not find the driver. This can differ on how OpenStack has been installed on your machine. With devstack the drivers are on: /opt/stack/cinder/cinder/volume/drivers . With packstack they will be on: /usr/lib/python2.7/site-packages/cinder/volume/drivers .

There was one last head ache that needed to be resolved to allow full integration of our cinder driver. When the driver is placed the correct directory, we proceed to add the necessary options (as shown below) to the /etc/cinder/cinder.conf file.

# first we need to enable the backend (lvm is already set by default)
enabled_backends = lvmdriver-1,foo
# then add these options to your driver configuration at the end of the file
[foo]
volume_backend_name = foo # this is super important!!!
volume_driver = cinder.volume.drivers.foo.FooDriver # path to your driver
# also add the options that you can set freely (volume_opts)
foo_api_endpoint = 'http://127.0.0.1:12956'

You must to set the volume_backend_name because it links Cinder to the correct backend, without it nothing will ever work (NOTHING!).

Finally, when you want to execute operations on it, you must create the volume type for your Cinder driver:

Now restart the Cinder services (c-vol, c-sch, c-api) and you should be able to use your own storage system through cinder.

]]>https://blog.zhaw.ch/icclab/how-to-write-a-cinder-driver/feed/015th OpenStack Meetuphttps://blog.zhaw.ch/icclab/15th-openstack-meetup/
https://blog.zhaw.ch/icclab/15th-openstack-meetup/#respondFri, 07 Apr 2017 15:30:42 +0000https://blog.zhaw.ch/icclab/?p=11324Continue reading →]]>On the 21st of March we held the 15th OpenStack meetup. As ever, the talks were interesting, relevant and entertaining. It was kindly sponsored by Rackspace and held at their offices in Zürich. Much thanks goes to them and to previous sponsors!

The two presentations were followed up by the panel discussion involving those that attended including our own Seán Murphy and was moderated by Andy Edmonds. Finally, as is now almost a tradition, we had a very nice apero!

Looking forward to the next and 16th OpenStack meetup!

]]>https://blog.zhaw.ch/icclab/15th-openstack-meetup/feed/0Openstack checkpointing is simplifiedhttps://blog.zhaw.ch/icclab/openstack-checkpointing-is-simplified/
https://blog.zhaw.ch/icclab/openstack-checkpointing-is-simplified/#respondThu, 26 Jan 2017 13:47:15 +0000https://blog.zhaw.ch/icclab/?p=11119Continue reading →]]>At ICCLab, we have recently updated the OpenstackOVA onboarding tool to include an exporting functionality that can help operators migrate and checkpoint individual VMs. Furthermore, researchers can now export VMs to their local environments, even use them offline, and at any time bring them back to the cloud using the same tool.

The OpenStack OVA onboarding tool automatically transforms selected virtual machines into downloadable VMDK images. Virtual machines and their metadata are fetched from OpenStack’s Nova service, and made packed as OVA file. The tool offers a GUI integration with OpenStack’s Horizon Dashboard, but can be also deployed separately.

Openstack onboarding tool supports:

Virtual machine export

Security group export (port forwarding rules for NAT interface)

Network Export (all networks are exported as internal networks)

Limitations:

QCOW2 images have the same size as instance disk space. It means that download image of X0 GB will take some time.

Network configuration of virtual machine in openstack deployment should be configured for non cloudinit setup too, otherwise the exported image will not have proper networking within Virtualbox or similar virtualization tool.

All NAT ports are required to have the network called “private” in the tenant.

We’ve prepared a short video about this tool’s deployment and its installation steps.

]]>https://blog.zhaw.ch/icclab/openstack-checkpointing-is-simplified/feed/0Monitoring an Openstack deployment with Prometheus and Grafanahttps://blog.zhaw.ch/icclab/monitoring-an-openstack-deployment-with-prometheus-and-grafana/
https://blog.zhaw.ch/icclab/monitoring-an-openstack-deployment-with-prometheus-and-grafana/#commentsThu, 24 Nov 2016 09:34:31 +0000https://blog.zhaw.ch/icclab/?p=10912Continue reading →]]>Following our previous blog post, we are still looking at tools for collecting metrics from an Openstack deployment in order to understand its resource utilization. Although Monasca has a comprehensive set of metrics and alarm definitions, the complex installation process combined with a lack of documentation makes it a frustrating experience to get it up and running. Further, although it is complex, with many moving parts, it was difficult to configure it to obtain the analysis we wanted from the raw data, viz how many of our servers are overloaded over different timescales in different respects (cpu, memory, disk io, network io). For these reasons we decided to try Prometheus with Grafana which turned out to be much easier to install and configure (taking less than an hour to set up!). This blog post covers the installation process and configuration of Prometheus and Grafana in a Docker container and how to install and configure Canonical’s Prometheus Openstack exporter to collect a small set of metrics related to an Openstack deployment.

Note that minor changes to this HOWTO are required to install theses services in a VM or in a host machine when using containers is not an option. As preparation, take note of your Openstack deployment’s locations for Keystone and the Docker host. Remember that all downloads should be verified by signature comparison for production use.

Installing and configuring Prometheus

First of all pull the Ubuntu image into you docker machine. Let’s call it docker-host.

Note that in this blog post we describe Prometheus installation process step-by-step – we chose to install it from scratch to get a better understanding of the system, but using the pre-canned Docker Hub image is also possible.

docker pull ubuntu:14.04

Then create the docker container opening the port 9090 which will be used to get/push metrics into Prometheus.

docker run -it -p 9090:9090 --name prometheus ubuntu:14.04

Inside the container download the latest version of Prometheus and uncompress it (version 1.3.1 is used in this HOWTO; the download size is ca. 16 MB).

Similarly, install and configure the Prometheus Openstack exporterinanothercontainer. Note that this container needs to be set up manually as there are configuration files to be changed and Openstack libraries to be installed.

Alternatively you could use Prometheus’s Node exporter for more detailed metrics on node usage – this needs to be installed in the controller/compute nodes and the prometheus.yml configuration file also needs to be changed. A docker container is also available at Docker Hub.

Although Prometheus provides some rudimentary graph support, combining it with a more powerful graphing solution makes it much easier to see what’s going on in your system. For this reason, we set up Grafana.

Installing Grafana

The latest version of Grafana (currently 4.0.0-beta2) had a lot of improvements in its user interface, it also supports now alerting and notifications for every panel available – refer to the documentation for more information. Its integration with Prometheus is very straightforward, as described below.

First of all, pull the grafana image into your docker-hostand create the docker container opening the port 3000 used to access it.

Visit http://docker-host:3000 and use the credentials admin/admin to log into the dashboard. In the Data Sources tab, add a new corresponding data source.

Create a new dashboard and add panels containing graphs using the Prometheus datasource.

Play around with metrics available and create your own dashboard! See a simple example below.

Conclusion

Although not many metrics are available yet to monitor an Openstack deployment the combination of Prometheus and Grafana is quite powerful for visualising data; also it was much easier to set up in comparison with Monasca. Further, from a cursory glance, Prometheus seems to be more flexible than Monasca and for these reasons it appears more promising. That said, we are still looking into Prometheus and how it can be used to properly understand resource consumption in an Openstack context, but that will come in another blog post!

]]>https://blog.zhaw.ch/icclab/monitoring-an-openstack-deployment-with-prometheus-and-grafana/feed/18Installing Monasca – a happy ending after much sadness and woehttps://blog.zhaw.ch/icclab/installing-monasca-a-happy-ending-after-much-sadness-and-woe/
https://blog.zhaw.ch/icclab/installing-monasca-a-happy-ending-after-much-sadness-and-woe/#commentsMon, 21 Nov 2016 13:12:13 +0000https://blog.zhaw.ch/icclab/?p=10904Continue reading →]]>In one of our projects we are making contributions to an Openstack project called Watcher, this project focuses on optimizing resource utilization of a cloud according to a given strategy. As part of this work it is important to understand the resource utilization of the cloud beforehand in order to make a meaningful contribution. This requires collection of metrics from the system and processing them to understand how the system is performing. The Ceilometer project was our default choice for collecting metrics in an Openstack deployment but as work has evolved we are also exploring alternatives – specifically Monasca. In this blog post I will cover my personal experience installing Monasca (which was more challenging than expected) and how we hacked the monasca/demo docker image to connect it to our Openstack deployment.

In our earlier series of blog posts we explained how to install, configure and test Monasca step-by-step using git, Maven and pip – a good option for learning the moving parts of the giant beast that is Monasca and how it is all linked together. This content was posted over a year ago and hence it is now a little bit stale; since that time work has been done on deployment using automation tools such as Ansible or Puppet. Generally, when working on this, we encountered significant issues with lack of documentation, unspecified versioning issues, unclear dependencies etc.

Our first attempt to install Monasca was based on different Ansible repositories available on Github – pointed to by the official page of Monasca. Although most of the repositories are Ansible roles, some try to put everything in a single place. Interestingly, many of those assume that Openstack is deployed with Devstack (which was not true in our case – we had a small Packstack deployment) and this caused many steps to fail during the installation as files were not where they were expected to be (in devstack many files are into /opt/stack/). Even after solving these issues Monasca itself did not seem to work and we always ended up with a system where only few monasca services were up and running. We’re not certain where the problems were, but there were multiple issues with Kafka and in some cases it was not possible to get monasca-agent or monasca-notifications working.

A tip if you want to go this path: Most of the repos use Ansible 1.9.2, which is not obvious. We had many issues using Ansible >= 2.0 and some roles using version 1.9.2 which required modification to the source code as there were changes in how loops are defined.

After the frustrating experience with Ansible we then decided to get back to basics, there is a functional out-of-the-box docker image in Docker Hub used for testing Monasca. It contains pretty much everything Monasca needs for running – even a local Keystone, Nova and Glance. We then decided to modify Monasca components in this image in order to connect it to our to Openstack cluster. Note that only Openstack credentials were modified inside the container – InfluxDB, MySQL and other services credentials also provided in the container were not modified. After relatively little tinkering, we managed to get this working as described below.

First of all, pull the image into your docker machine – let’s call it docker-host. Note that docker-host must be able to access your Openstack deployment (via the public APIs) and vice-versa

docker pull monasca/demo

Next, create the docker container but it is necessary to modify its entrypoint, so it does not start setting up monasca automatically and we can change the Monasca configuration files; also, note that the since it will not be running Keystone, we will remove the Keystone port:

Change the value of port_to_horizon in case port 80 is already used in docker-host.

Inside the docker container you will be in the ‘/setup’ directory, in this directory you will see the ansible files used to set up this container. The main file in this directory is called demo-start.sh, this file will start all the services required by Monasca but before running this script it is necessary to modify the following Monasca Openstack configuration details.

In /etc/monasca/api-config.yml add your credentials in the middleware section (note that you will need to install a text editor in the container):

In /setup/demo-start.sh there is an error regarding the location of monasca-notification init script, modify the ‘/usr/local/bin/monasca-notification &’ (line 44) to ‘/opt/monasca/bin/monasca-notification &’. Also remove the playbook which installs the Keystone service (line 11):

Finally remove services which are not necessary anymore, eg the local Keystone, from start.sh, leave apache2 and change the variable OPENSTACK_HOST in /etc/openstack-dashboard/local_settings.py to the IP where Keystone is installed:

Once all of this is done, you can now run /setup/demo-start.sh to set everything up – it may take a couple of minutes. When it has finished, you can visit your monasca-dashboard page at http://docker-host:{port_to_horizon}. Log in as the user Monasca and visit the monitoring tab in the dashboard.

In this tab you will be able to create alarms definitions and notifications. You will also be able to see an overview of the current state of the system in the Grafana dashboard. To collect more data regarding your Openstack cluster you will need to install and configure monasca-agent service in each of the controller/compute nodes. It is possible to do it in a virtualenv so it does not modify any library in the system.

Then you’ve got a fully functional Monasca system which works with your Openstack cluster – it’s a good solution for experimenting with Monasca to understand how powerful it really is. This solution is probably not the best solution for a production deployment, however – it would probably make more sense to decouple the services and put the data in a persistent data store, but that’s another day’s work!

Troubleshooting

If you are getting “requests.exceptions.ConnectionError: (‘Connection aborted.’, gaierror(-2, ‘Name or service not known’))” error when running demo-start.sh script that means the container is not able reach the endpoint defined in Openstack for the monitoring service. Make sure that the container is able to connect to the external interface in your Openstack cluster and the Keystone endpoint for the monitoring service is configured correctly.

The demo-start.sh is the main script in this container, in case you find any issues you can execute each of the commands in this script sequentially to understand more clearly what is happenning.

“Exception in thread “main” java.lang.RuntimeException: Topology with name `thresh-cluster` already exists on cluster”. This error is known to happen often as the thresh-cluster service tries to configure itself for a second time, you can ignore it.

]]>https://blog.zhaw.ch/icclab/installing-monasca-a-happy-ending-after-much-sadness-and-woe/feed/16Integration of Openstack OVA importing tool to Horizonhttps://blog.zhaw.ch/icclab/integration-of-ova-importing-tool-to-horizon/
https://blog.zhaw.ch/icclab/integration-of-ova-importing-tool-to-horizon/#respondFri, 14 Oct 2016 15:33:07 +0000https://blog.zhaw.ch/icclab/?p=10703Continue reading →]]>ICCLab is announcing an integration of the OpenstackOVA onboarding tool into OpenStack’s Horizon dashboard. To deploy the OVA file to Openstack all images are extracted from the file and uploaded to the Openstack cluster, all necessary file format transformations are automatically performed, glance images get created and the tool creates a heat stack out of them. As we mentioned a couple of weeks ago, uploading your local VMs into OpenStack was never easier.

Now we are making it even better by allowing everyone to import their OVA files using Horizon dashboard. In order to make the deployment straightforward and easy on future releases of Horizon, the integration itself is implemented as a separate view.

Once you (or someone from operations) have deployed it, simply navigate to the Onboarding tab where you will see a table view of all created stacks via the Onboarding tool. Due to this integration, you no longer need to provide credentials when onboarding VMs, as they are retrieved from your session automatically.

We’ve prepared a short video about this tool’s deployment and its installation steps.

]]>https://blog.zhaw.ch/icclab/integration-of-ova-importing-tool-to-horizon/feed/0A new tool to import OVA Applications to Openstackhttps://blog.zhaw.ch/icclab/openstack-ova-onboarding-tool-release/
https://blog.zhaw.ch/icclab/openstack-ova-onboarding-tool-release/#commentsThu, 01 Sep 2016 14:25:04 +0000https://blog.zhaw.ch/icclab/?p=10563Continue reading →]]>If you ever thought of uploading your local VMs to OpenStack, perhaps you have come across OpenStack’s support for importing single virtual disk images. However, this cannot be used to deploy complicated VM setups, including network configurations and multiple VMs connected to each other.We at ICCLab have therefore decided to develop a tool that will allow anyone to upload their VM setups from their local environments directly to OpenStack. We call it OpenStack VM onboarding tool and it’s available as open source.

VM onboarding tool features:

Easy to run – the tool comprises of simple frontend, backend and Openstack client libraries to access Openstack APIs. All these components can be easily run with one command.

Easy to Import – to import an OVA file the user needs to provide only the basic Openstack credentials (username, password, tenant, region, keystone URL) and an OVA file.