Category Archives: Technical

VMware Integrated OpenStack(VIO) announced the official support for Barbican, OpenStack secrets manager, in version 5.1. With Barbican, cloud Operators can offer Key Management as a service by leveraging Barbican API and command line(CLI) to manage X.509 certificates, keys, and passwords. Basic Barbican workflow is relatively simple – invoke the secrets-store plugin to encrypt a secret on the store and decrypt a secret on retrieval. In addition to generic secrets management, some OpenStack projects integrate with Barbican natively to provide enhanced security on top of its base offering. This blog will introduce Barbican consumption and operation maintenance through the use of Neutron Load Balancer as a Service (LBaaS).

Understanding Policies

Barbican scopes the ownership of a secret at the OpenStack project level. For each API call, OpenStack will check to ensure the project ID of the token matches the project ID stored as the secret owner. Further, Barbican uses roles and policies to determine access to secrets. Following roles are defined in Barbican::

Admin – Project administrator. This user has full access to all resources owned by the project for which the admin role is scoped.

Creator – Users with this role are allowed to create and delete resources. Users with this role cannot delete other user’s resources managed within same project. They are also allowed full access to existing secrets owned by the project in scope.

Observer – Users with this role are allowed access to existing resources but are not allowed to upload new secrets or delete existing secrets.

Audit – Users with this role are only allowed access to the resource metadata. So users with this role are unable to decrypt secrets

VIO 5.1 ships with “admin” and “creator” role out of the box. A project member must be assigned with the creator role to consume barbican. Based on the above roles, Barbican defines a set of rules or policies for access control. Only operations specified by the matching rule will be permitted.

While the policy framework works well, but secrets management is never one size fits all, and there are limitations with the policy framework if fine-grain control is required. Scenarios such as grant specific user access to a particular secret or upload a secret for which only the uploader has access needs OpenStack ACLs. Please refer to ACL API User Guide for full details.

Supported Plugin

The Barbican key manager service leverages secret-store plugins to allow authorized users to store secrets. VIO 5.1 supports two type of plugins, simple crypto and KMIP enabled. Only a single plugin can be active for a VIO deployment. Secret stores can be software-based, such as a software token, or hardware devices such as a hardware security module (HSM).

Simple crypto plugin

The simple crypto plugin uses a single symmetric key, stored locally on the VIO controller in the /etc/barbican/barbican.conf file to encrypt and decrypt secrets. This plugin also leverages local Barbican database and stores user secrets as encrypted blobs in the local database. The reliance on local text file and database for storage is considered insecure, and therefore upstream community considers simple crypto plugin to be suitable for development and testing workloads only.

Secret store KMIP plugins

The KMIP plugin stores secrets securely in an external KMIP-enabled device. The Barbican database, instead of storing encrypted secrets, maintain location references of secrets for later retrieval. Client certificate-based authentication is the recommended approach to integrate the plugin with the KMIP enabled device.

Example Barbican Consumption:

One of the most commonly requested use case specific to VIO is Barbican integration with Neutron LBaaS to offer HTTPS offload. This is a five step process, we will review each step in detail.

Install KMIP server (Greenfield only)

Integrate KMIP using VIOCLI

ACL update

Workflow to create secret

Workflow to create LBaaSv2

Please note, you must leverage OpenStack API or CLI for step #4. Horizon support for Barbican is not available.

Install KMIP server

Production Barbican deployment requires a KMIP server. In a greenfield deployment, Dell EMC CloudLink is a popular solution VMware vSAN customers leverage to enable vSAN storage encryption. CloudLink includes both a key management server (KMS) as well as the ability to control, monitor and encrypt secrets across a hybrid cloud environment. Additional details on CloudLink is available from VMware solution exchange.

Integrate KMIP using VIOCLI

To integrate with CloudLink KMS or any other KMIP based secret store, simply login into the VIO OMS server and issue the following VIOCLI command;

Configure Barbican to use the KMIP plugin.

viocli barbican –secret-store-plugin KMIP

–user viouser \

–password VMware**** \

–host <KMIP host IP> \

–ca-certs /home/viouser/viouser_key_cert/ca.pem \

–certfile /home/viouser/viouser_key_cert/cert.pem \

–keyfile /home/viouser/viouser_key_cert/key.pem –port 5696

Successful completion of VIOCLI command performs following set of actions:

ACL updates based on consumption

Neutron LBaaS relies on a Barbican service account to read and push certificates and keys stored in the Barbican containers to a load balancer. The Barbican service user is an admin member of the service project, part of the OpenStack Local domain. Default Barbican security policy does not allow admin or member of one project to access secrets stored in a different project. In order for Barbican service user to access and push certificate and keys, tenant users must grant access to the service account. There are two ways to allow access:

Option 1:

1). Tenant creator gives Barbican service user access using the OpenStack ACL command. Cloud administrator needs to supply the UUID of the Barbican service account.

Repeat this command with each certificate, key, and container you want to provide Neutron access to.

Option 2:

2.) If cloud administrators are comfortable providing Neutron with access to secrets without users granting access to individual objects, cloud administrators may elect to modify the Barbican policy file. Implementing this policy change means that tenants won’t need to add the Neutron barbican service_user to every object, which makes the process of creating TERMINATED_HTTPS listeners easier. Administrators should understand and be comfortable with the security implications of this action before implementing this approach. To perform the policy change, use a custom-playbook to change the following line in the Barbican policy.json file:

From: “secret:get”: “rule:secret_non_private_read or rule:secret_project_creator or rule:secret_project_admin or rule:secret_acl_read”,

To: “secret:get”: “rule:secret_non_private_read or rule:secret_project_creator or rule:secret_project_admin or rule:secret_acl_read or role:admin”,

Workflow to Create Secret:

This step assumes you have pre-created certificates and keys. If you have not created keys and certificates before, please refer to this blog for details. To follow steps outlined below, make sure to name your output files accordingly (server.crt and server.key). To upload a certificate:

openstack secret store –name=’certificate’ \

–payload=”$(cat server.crt)” \

–secret-type=passphrase

Most of options are fairly self-explanatory, passphrase indicates a plain text. Repeat the same command for keys:

openstack secret store –name=’private_key’ \

–payload=”$(cat server.key)” \

–secret-type=passphrase

you can confirm by listing out all secrets:

Final, create a TLS container pointing to both private key and certificate secrets:

Workflow to create LBaaSv2

With Barbican service up and running, ACL configured to allow retrieval of secret keys, let’s start to create a Load balancer and upload a certificate and key from the KMS server. Load balancer creation workflow does not change with Barbican. When creating a listener, be sure to specify TERMINATED_HTTPS as the protocol, and URL of the TLS container stored in Barbican.

Please note:

If you are testing Barbican against NSX-T, NSX-MGR must be running at least version 2.2 or higher.

Written by Xiao Gao, with valuable feedbacks and inputs from Mark Voelker.

While working with customers that are switching over to VMware Integrated OpenStack (VIO) from a different OpenStack distribution, customers expressed the need to update policies. Reasons were:

Backward compatibility with their legacy OpenStack deployment.

Internal company process and procedure alignment.

While updating policy is not any more complicated on VIO when compared to other distributions, it is an operation that we traditionally advised our customers to avoid. Following are some of the reasons:

1). Upgrade. While many non-default changes can seem trivial and straightforward, VMware can’t guarantee upstream implementation will always be backward compatible when moving between releases. Therefore, responsibility of maintaining day-2 changes lies within the customer

2). Snowflake avoidance. Upstream gate tests focus almost exclusively on default policies. The risk of exposing unexpected side effect increases when the security posture of an operation is relaxed or tightened. Security is also a concern when relaxing policies. Similarly, most popular OpenStack orchestration/monitoring tools such as Terraform, Gophercloud, or Nagios are implemented assuming default policies. When policies are made more restrictive, it can cause your favorite OpenStack tools to fail.

Snowflakes not only are difficult to support and maintain, often cause of unexpected outages.

3). Leverage external CMP for enhanced governance and control. External CMP such as the vRA is designed to integrate business processes into IAAS consumption. Instead of maintaining low-level policies changes, leverage out of box capabilities of vRA to control what users will have access to.

Implementation Options:

We understand there are scenarios where policy changes are required. Our recommendation for those scenarios is to leverage VIO custom playbook to make those changes. The basic idea behind custom playbook:

Customer will code up what has to change using Ansible.

VIO will decide when to make required changes, to survive upgrades and other maintenance tasks.

While VIO doesn’t sanction contents of the custom playbook, it’s essential to write the playbook in a manner that is modular and agnostic to the OpenStack version. Ideal playbook should be stateless, grouped based on operational action, and not restrictive towards alignment with the upstream (see example section for details). Loggings is on by default.

Working Example:

Let’s look at an example. Say we want regular users to be able to create shared networks. To do that we need to modify /etc/neutron/policy.json and change:

“create_network:shared”: “rule:admin_only”

to:

“create_network:shared”: “”

There is number of ways to accomplish above task. You can go down the path of j2 templates and introduce variables for each policy modification. But this approach requires discipline from the operator to update his/her entire set of j2 policy template(s) before any significant upgrade to avoid drift or conflicts with upstream. On the other hand, if you leverage direct file manipulation method, you will change only parameters that are required in your local environment, and leave everything else in constant alignment with upstream.

example uses back references (e.g., the parenthesis in the “regex” line and the \\1 and \\2 in the “line” line) to preserve the indentation/leading spaces on the beginning of each line and the comma at the end of the line (if it’s present). Back reference makes the regex a tad more complicated-looking, and it keeps the formatting in place.

Log Outputs:

Below are sample logs:

Conclusion

This post outlined thought processes involved when updating OpenStack Policies. I would love to hear back from you.

Also, VIO 4.1 is now GA. You can Download a 60-day VIO evaluation now and get started.

A production cloud isn’t very efficacious unless users have the ability to run virtual machine images required by their application. A cloud image is a single file that contains a virtual disk that has an operating system. For many organizations, the simplest way to obtain a virtual machine image is to download a prebuilt base cloud image with a pre-packaged version of cloud-init to support user-data injection. Once downloaded, an organization would leverage tools such as Packer to further customize and harden on top of the base image before rolling to production. Most operating system projects and vendors maintain official images for direct download. Openstack.org maintains a list of most commonly used images here.

Recently we received some queries about the proper way to import prebuilt QCOW2 native cloud images into VMware Integrated OpenStack. Images import correctly, but would not successfully boot. Common symptoms are “no Operating System found” messages generated by the virtual machine’s BIOS, the guest OS hanging during the boot cycle, or DHCP failure when trying to acquire an IP address. After further analysis, problems were either caused by older upstream tooling or simple adjustments required in the cloud image to match the vSphere environment. Specifically:

Some storage vendors need StreamOptimized image format.

Guest Images are attempting to write boot log to ttyS0, but the serial interface is not available on the VM.

Defects in earlier versions of the qemu-img tool while creating streamOptimized images.

DHCP binding failure caused by Predictive Network Interface Naming.

To overcome these issues, we came up with the following set of best practices to help you simplify the image import process. I thought it would be a good idea to share our recommendations so others can avoid running into similar issues.

1). VIO 3.x and earlier, serial console output is not enabled. When booting an image that requires serial console support, use libguestfs to edit the grub.cfg and remove all references to “console=ttyS0”. Libguestfs provides a suite of tools for accessing and editing VM disk images. Once installed the “guest mount” command-line tool can be used to mount qcow2 based images. By default, the disk image mounts in read-write mode. More info on Libguestfs here.

# guestmount -a xxx-cloudimg-amd64.img -m /dev/sda1 /mnt

# vi /mnt/boot/grub/grub.cfg

# umount mnt

See below screen Capture:

2). VMware vSAN requires all images to be in streamOptimized format. When converting to VMDK format, use the –o flag to specify the subformat as streamOptimized:

“lsilogic” is the recommended adapter type. Although it is possible to set the adapter type during image upload into glance, we recommend as a good practice to always set the adapter type as part of the image conversion process.

Older versions of the qemu-img tool contain a bug that causes problems with the streamOptimized subformat. The following command can be run after converting an image to correct the problem: printf ‘\x03’ | dd conv=notrunc of=xxx-server-cloudimg-amd64.vmdk bs=1 seek=$((0x4)). It is harmless to execute the printf even if you’re using a version of qemu-tools that has the fix: all the command does is set the VMDK version to “3” which correct version of qemu-img will already have done. If you are not sure what version of qemu-tools you have, apply the printf command.

3). In the case of CentOS, Udev rule ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules as part of the image bundle is ignored during CentOS image boot up and Predictable Network Interface Naming is enabled as a result. Our recommendation is to disable predictive naming using grub. You can find more information on my previous blog.

4). Finally, with Cirros QOCW image, preserve the adapter type as ‘ide’ during the QCOW2 to VMDK conversion process. There’s currently an upstream bug open.

Once converted, you can look into the image metadata and validate information such as disk and image type before uploading into Glance image repository. Image metadata can be viewed by display the first 20 lines of the VMDK

# cat xxx-server-cloudimg-amd64.vmdk | head -20

You can add the newly converted image into glance using OpenStack CLI or Horizon. Set the public flag when ready for end user consumption.

OpenStack CLI:

Horizon:

Your cloud is as useful as the application and virtual machine images you can support. By following above simple best practice guidelines, you will deliver a better user experience to your end users by offering more Virtual machine varieties with significantly reduced lead time.

Visit us at VMworld in Las Vegas; we have a large number of Demo and speaking sessions planned:

Resource and over-subscription management are always the most challenging tasks facing a Cloud Admin. To deliver a guaranteed SLA, one method OpenStack Cloud Admins have used is to create separate compute aggregates with different allocation / over-subscription ratios. Production workloads that require guaranteed CPU, memory, or storage would be placed into a non-oversubscribed aggregate with 1:1 over-subscription, dev workloads may be placed into a best effort aggregate with N:1 over-subscription. While this simplistic model accomplishes its purpose of an SLA guarantee on paper, it comes with a huge CapEx and/or high overhead for capacity management / augmentation. Worst yet, because host aggregate level over-subscription in OpenStack is simply static metadata consumed by the nova scheduler during VM placement, not real time VM state or consumption, huge resource imbalances within the compute aggregate and noisy neighbor issues within a nova compute host are common occurrences.

New workloads can be placed on a host running close to capacity (real time consumption), while remaining hosts are running idle due to differences in application characteristics and usage pattern. Lack of automated day 2 resource re-balance(management) further exacerbates the issue. To provide white glove treatment to critical tenants and workloads, Cloud Admins must deploy additional tooling to discover basic VM to Hypervisor mapping based on OpenStack project IDs. This is both expensive and ineffective in meeting SLAs.

Over-subscription works if resource consumption can be tracked and balanced across a compute cluster. Noisy neighbor issues can be solved only if the underlying infrastructure supports quality of service (QoS). By leveraging OpenStack Nova flavor extra-spec extensions along with vSphere industry proven per VM resource reservation allocation (expressed using shares, limits and reservations), OpenStack Cloud Admins can deliver enhanced QoS while maintaining uniform consumption across a compute cluster. It is possible to leverage Image metadata to deliver QoS as well, this blog will focus on Nova flavor extra-spec.

The VMware Nova flavor extension to OpenStack was first introduced upstream in Kilo and is officially supported in VIO release 2.0 and above. Additional requirements are outlined below:

Share Level – The allocation level. This can be ‘custom’, ‘high’ ‘normal’ or ‘low’.

Shares Share – In the event that ‘custom’ is used, this is the number of shares.

Complete Nova flavor extra-spec details and deployment options can be found here. vSphere Resource Management capabilities and configuration guidelines is a great reference as well and can be found here.

Let’s look at an example using Hadoop to demonstrate VM resource management with flavor extra-specs. Data flows from Kafka into HDFS, every 30 minutes there’s a batch job to consume the newly ingested data. Exact details of the Hadoop workflow are outside the scope of this blog. If you are not familiar with Hadoop, some details can be found here. Resources required for this small scale deployment are outlined below:

Node Type

Core

(reserved – Max)

Memory

(reserved – Max)

Disk

Network Limit

Master / Name Node

4

16 G

70 G

500 Mbps

Data Node

4

16 G

70 G

1000 Mbps

Kafka

0.4-2

2-4 G

25 G

100 Mbps

Based on above requirements, Cloud Admin needs to create Nova flavors to match maximum CPU / Memory / Disk requirements for each Hadoop component. Most of OpenStack Admins should be very familiar with this process.

Based on the reservation amount, attach corresponding nova extra specs to each flavor:

Once extra specs are mapped, confirm setting using the standard nova flavor-show command:

In just three simple steps, resource reservation settings are complete. Any new VM consumed using new flavors from OpenStack (API, command line or Horizon GUI) will have resource requirements passed to vSphere (VMs can be migrated using the nova rebuild VM feature).

Instead of best effort, vSphere will guarantee resources based on nova flavor extra-spec definition. Specific to our example, 4 vCPU / 16G / Max 1G network throughput will be reserved for each DataNode, NameNode with 4 vCPU / 16G / Max 500M throughput and Kafka nodes will have 20% vCPU / 50% Memory reserved. Instances boot into “Error” state if requested resources are not available, ensuring existing workload application SLAs are not violated. You can see that the resource reservation created by the vSphere Nova driver are reflected in the vCenter interface:

Name Node CPU / Memory:

Name Node Network Bandwidth:

Data Node CPU / Memory:

Data Node Network Bandwidth:

Kafka Node CPU / Memory:

Kafka Network Bandwidth:

vSphere will enforce strict admission control based on real time resource allocation and load. New workloads will be admitted only if SLA can be honored for new and existing applications. Once a workload is deployed, in conjunction with vSphere DRS, workload rebalance can happen automatically between hypervisors to ensure optimal host utilization (future blog) and avoid any noisy neighbor issues. Both features are available out of box, no customization is required.

By taking advantage of vSphere VM resource reservation capabilities, OpenStack Cloud Admins can finally enjoy the superior capacity and over-subscription capabilities a cloud environment offers. Instead of deploying excess hardware, Cloud Admins can control when and where additional hardware are needed based on real time application consumption and growth. Ability to consolidate, simplify, and control your infrastructure will help to reduced power, space, and eliminate the need for any out of box customization in tooling or operational monitoring. I invite you to test out nova-extra spec in your VIO environment today or encourage your IT team to try our VMware Integrated OpenStack Hands-On-Lab, no installation is required.

This article describes an OpenStack setup. To put it into context, I recently joined VMware as a Technical Marketing Manager responsible for VMware Integrated OpenStack technical product marketing and field enablement. After a smooth onboarding process (accounts, benefits, etc), my first task was to get lab environments I inherited up and running again. I came from a big OpenStack shop where deployment automation was built in house using both Puppet and Ansible, so my first questions were; Where’s the site hiera data? Settings for Cobbler environment? Which playbook to run to spin up controller VM nodes? Once the vm is up, which playbooks are responsible for deployment and configuration of keystone, nova, cinder, neutron, etc. How to seed the environment once OpenStack is configured and running? It was always a multi-person, multi-week effort. Everyone maintained a cheat sheet loaded with deployment caveats and workarounds (long list since increased feature = increased complexity = increased caveats = slow time to market). Luckily with VMware Integrated OpenStack, all those decision points and complexity are abstracted for the Cloud Admin. Really, the only decision to make is:

Restore from database backup – Restore from backup processes can be found here.

Redeploy and Import.

Redeploy seemed more interesting so I decided to give it a shot (will save restore for a different blog if there is interest).

Note: Redeploy will not clean up vSphere resources. One could re-import vm resources from vCenter once deployment completes. Refer to Instructions here.

One instance of NSX-v spanning across all 3 clusters from a networking perspective.

VIO OpenStack Deployment:

VMware Integrated Openstack can be deployed in 2 ways, Full HA or Compact. Compact mode was recently introduced in 3.0 and requires significantly fewer hardware resources and memory than full HA mode. All OpenStack components can be deployed in 2 VMs with Compact mode. Enterprise grade OpenStack high availability can be achieved by taking advantage of the HA capabilities of the vSphere infrastructure. Compact mode is useful for multiple, small deployments. Full HA mode provides high availability at the application layer. The HA deployment consists of a dedicated management node (aka OMS node), 3 DB, 2 HAproxy and 2 Controller nodes. Ceilometer can be enabled after completion of the VIO base stack deployment. Enterprise HA mode is what I chose to deploy.

Since this environment is new to me, I wanted to avoid playing detective and spending days trying to reverse engineer the entire setup. Luckily VIO has a built-in export configuration capability. Simply navigate to the OpenStack deployment (Home -> VMware Integrated OpenStack -> OpenStack Deployments), all OpenStack settings can be exported with a single click:

In some ways the exported configuration file is similar to traditional site hiera data except much simpler to consume. The data is in JSON format so it is easy to read and only information specific to my OpenStack setup is included. I no longer need to worry about kernel / TCP settings, interfaces to configure for management vs. data, NIC bonding, driver settings, haproxy endpoints, etc. Even the Ansible inventory is automatically created and managed based on deployment data. This is because VIO is designed to work out of box with the VMware suite of products, allowing Cloud Admins to focus on delivering SLAs instead of maintaining thousands of hard to remember key/value pairs. Advanced users can still look into inventory and configuration parameter details. The majority of deployment metrics are maintained on the OMS node in the following directories: (Please Note: The settings below are intended for viewing only and should not be modified without VMware support. The OMS node is primarily used as a starting point for troubleshooting, run viocli commands and SSH to the other nodes):

With configuration saved, I went ahead and deleted the old deployment and clicked the Deploy OpenStack link to redeploy:

The process to re-deploy a VIO OpenStack cluster is extremely simple, one simply selects an exported template to pre-fill configuration settings.

The remaining deployment processes are well documented via other VMware blogs. References can be found here.

The entire Full HA mode deployment process took slightly over 50 minutes because of an unexpected NSX disk error that prevented neutron from starting. The deployment took 30 minutes with a clean environment (see below). Compact mode users should expect deployment to take as little as 15 minutes.

Create VM and Test External Connectivity:

Once deployment is completed, simply create a L2 private network, and test that VMs can boot successfully in the default tenant project. Note, in order for a VM to connect externally, an external network needs to be created, associate the private and external networks to a NAT router, request a floating ip and finally associate the floating ip to the test VM. This is all extremely simple as VIO is 100% based on DefCore compliant OpenStack APIs on top of VMware’s best-of-breed SDDC infrastructure. Two neutron commands are all that’s needed to create an external network:

Floating IP is a fancy OpenStack word for NAT. In most OpenStack implementations NAT translation happens in the neutron tenant router. A VIO neutron tenant router(s) can be deployed in 3 different modes – centralized exclusive, centralized shared, or distributed. Performance aside, the biggest difference between centralized and distributed mode is the number of control VM’s deployed to support the logical routing function. Distributed mode requires 2 NSX control plane router VMs, one router instance (dLR) for optimized East to West traffic between hypervisors. A second instance is deployed to take care of North – South external traffic flow via NAT. A single NSX control VM instance is required for centralized mode. In centralized mode, all routed traffic ( N -> S, E -> W) flows through a central NSX Edge Service Gateway (ESG). Performance and scale requirements will determine which mode to choose. Centralized shared is the default behavior. Marcos Hernandez had written an excellent blog in the past on NSX networking and VIO. Marcos’s blog can be found here.

An enterprise grade NSX NAT router can be created via three neutron commands, one command to create the router, and two commands to associate corresponding neutron networks.

Once the router is created, simply allocate a floating IP, and associate the floating IP to the test VM instance:

The entire process from deployment to external VM reachability took me less than 3 hours in total, not bad for a new guy!

Deploying and configuring a production grade OpenStack environment traditionally takes weeks by highly skilled DevOps engineers specializing in deployment automation, in-depth knowledge of repo and package management, and strong Linux system administration. To come up with the right CI/CD process to support new features and align with upstream within a release is extremely difficult and results in snowflake environments.

The VIO approach changes all that. I’m impressed with what VMware has done to abstract away traditional complexities involved in deploying, supporting, and maintaining an enterprise grade OpenStack environment. Leveraging VMware’s suite of products in the backend, an enterprise grade OpenStack can be deployed and configured in hours rather than weeks. If you haven’t already, make sure to give VMware Integrated OpenStack a try, you will save tremendous amounts of time, meet the most demanding requests of application developers, while providing the highest SLA possible. Download now and get started , or dare your IT team to try our VMware Integrated OpenStack Hands-On-Lab , no installation required.

During Wednesday morning’s keynote session at the OpenStack Summit in Barcelona, I will be on stage along with several other vendors to show off some of our latest work on interoperability between OpenStack clouds. We will be demonstrating a single workload running successfully on over a dozen different vendors’ OpenStack products without modification, including VMware Integrated OpenStack 3.0. The idea got started back in April at the last OpenStack Summit when our friends at IBM challenged vendors to demonstrate that their products were interoperable publicly.

VMware has long been a proponent of fostering interoperability between OpenStack clouds. I currently co-chair the Interop Working Group (formerly known as the DefCore Committee), and VMware Integrated OpenStack 3.0 is an approved OpenStack-Powered product that is compliant with the 2016.08 interoperability guideline, the newest and strictest guideline approved by the OpenStack Foundation Board of Directors. We also helped produce the Interop Working Group’s first ever report on interoperability issues. So why do we care about interoperability? Shouldn’t everything built on OpenStack behave the same anyhow? Well, to quote the previously mentioned report on interoperability issues:

“OpenStack is tremendously flexible, feature-rich, powerful software that can be used to create clouds that fit a wide variety of use cases including software development, web services and e-commerce, network functions virtualization (NFV), video processing, and content delivery to name a few. Commercial offerings built on OpenStack are available as public clouds, installable software distributions, managed private clouds, appliances, and services. OpenStack can be deployed on thousands of combinations of underpinning storage, network, and compute hardware and software. Because of the incredible amount of flexibility OpenStack offers and the constraints of the many use cases it can address, interoperability between OpenStack clouds is not always assured: due to various choices deployers make, different clouds may have some inconsistent behaviors. One of the goals of the [Interop Working Group]’s work is to create high interoperability standards so that end users of clouds can expect certain behaviors to be consistent between different OpenStack-Powered Clouds and products.”

Think of it this way: another amazingly flexible, powerful thing we use daily is electricity. Electricity is pretty much the same stuff no matter who supplies it to you or what you are using it for, but the way you consume it might be different for different use cases. The outlet I plug my laptop into at home is a different shape and supplies a different voltage than the one my electric oven is connected into since the oven needs a lot more juice to bake my cookies than my laptop does to type up a blog post. My home’s air conditioner does not even have a plug: it is wired directly into the house’s circuit breaker. I consume most of my electricity as a service provided by my power company, but I can also generate some of my power with solar panels I own myself as long as their outputs can are connected to my power grid. Moreover, to power up my laptop here in Barcelona, I brought along a plug adapter since Europe has some differences in their power grid based on their set of standards and requirements. However, even though there are some differences, there are many commonalities: electricity is delivered over metal wiring, terminated at some wall socket, most of the world uses one of a few different voltage ranges, and you pay for it based on consumption. OpenStack is similar: An OpenStack deployment built for NFV workloads might have some different characteristics and interfaces exposed than one made as a public compute cloud.

What makes the Interop Challenge interesting is that it is complimentary to the work of the Interop Working Group in that it looks at interoperability in a slightly different light. To date, the Interop Working Group has mostly focused its efforts on API-level interoperability. It does so by ensuring that products bearing the OpenStack-Powered mark, pass a set of community-maintained Tempest tests to prove that they expose a set of capabilities (things like booting up a VM with the Nova v2 API or getting a list of available images using the Glance v2 API). Products bearing the OpenStack-Powered logo are also required to use designated sections of upstream code, so consumers know they are getting community-developed code driving those capabilities. While the Interop Working Group’s guidelines look primarily at the server side of things, the Interop Challenge seems to address a slightly different aspect of interoperability: workload portability. Rather than testing a particular set of API’s, the Interop Challenge took a client-side approach by running a real workload against different clouds—in this case, a LAMP stack application with a load-balanced web server tier and a database backend tier, all deployed via Ansible. The idea was to take a typical application with commonly-used deployment tools and prove that it “just works” across several different OpenStack clouds.

In other words, the guidelines produced by the Interop Working Group assure you that certain capabilities are available to end users (just as I can be reasonably confident that any hotel room I walk into will have a socket in the wall from which I can get electricity). The Interop Challenge compliments that by looking at a more valid use case: it verifies that I can plug in my laptop and get some work done.

Along the way, participants also hoped to begin defining some best practices for making workloads more portable among OpenStack clouds to account for some of the differences that are a natural side effect of OpenStack’s flexibility. For example, we found that the LAMP stack workload was more portable if we let the user specify certain attributes of the cloud he intended to use – such as the name of network the instances should be attached to, the image and flavor that should be used to boot up instances, and block device or network interface names that would be utilized by that image. Even though we will only be showing one particular workload on stage, that one workload serves as a starting point to help flesh out more best practices in the future.

On November 2nd, at 9 am PST/12 pm EST I will be hosting a free webinar, as part of our Getting More Out Of series, focused on OpenStack Integration with VMware NSX. We will also cover how easy it is to deploy VMware’s OpenStack distribution, VIO, and the Day-2 operational tools in the VMWare SDDC portfolio that make it possible for Cloud admins to properly monitor and optimize their OpenStack deployments.

As I mentioned in my 3-part blog series on Neutron-NSX integration, as more features are added to OpenStack (especially to Neutron), its architecture becomes more complex (a universal perception amongst OpenStack users). NSX mitigates these issues and provides a scalable and robust platform that incorporates Enterprise-grade network services into your OpenStack architecture.

Come and learn more about the benefits and participate in this unapologetically technical webinar, with live Q&A with our expert panel.

If you’re ready to try VIO, take it for a spin with the Hands on Labs that illustrates a step-by-step walkthrough of how to deploy OpenStack in Compact Management Mode in under fifteen minutes.

Deploying OpenStack challenges even the most seasoned, skilled IT organizations. With integrations, configurations, testing, re-testing, stress testing and more. For many, deploying OpenStack appears as an IT ‘Science Project’, wherein the light at the end of the tunnel dims with each passing month.

VMware Integrated OpenStack takes a different approach, reducing the redundancies and confusion of deploying OpenStack with the new Compact Management Control Pane. In the Compact Mode UI, wait minutes, not months. Enterprises seeking to evaluate OpenStack or those that are ready to build OpenStack clouds in the most cost efficient manner now have the ability to deploy in as little as 15 minutes quickly.

The architecture for VMware Integrated OpenStack is optimized to support compact architecture mode, reducing the need for support, overall resource costs, and the operational complexity keeping an enterprise from completing their OpenStack adoption.

The most recent update to VMware Integrated OpenStack focuses on the ease of use and the immense benefit to administrators – access and integration to the VMware ecosystem. The seamless integration of the family of VMware products allows the administrators to leverage their current VMware products to enhance their OpenStack, in combination with the ability to manage workloads through developer friendly OpenStack APIs.

The most recent update to VMware Integrated OpenStack focuses on the ease of use and the immense benefit to administrators – access and integration to the VMware ecosystem. The seamless integration of the family of VMware products allows the administrators to leverage their current VMware products to enhance their OpenStack, in combination with the ability to manage workloads through developer friendly OpenStack APIs.

Palo Alto Networks has one of the most mature and robust integrations with VMware NSX and we also share many joint customers in production. Together, we have seen tremendous success in the market, and that success can now extend to those prospects wanting to do OpenStack, while augmenting their security strategy with the added visibility and protection that Palo Alto offers.

The basic tenets for this integration between Palo Alto Networks and VMware NSX, in the context of an OpenStack deployment, remain the same:

The Security/Firewall Team is in complete control of the security lifecycle of the tenant apps.

Identify the Compute clusters that will host your OpenStack workloads and deploy Palo Alto network introspection to those clusters:

Ensure the Service VMs (local firewalls) are properly registered and licensed in Panorama:

Create an NSX Security Group with a classification criteria that meets your needs. In this example we are using the proverbial example based on VM Name (Name Contains Web in this case):

In Panorama, create a Dynamic Address Group for the OpenStack Instances, that corresponds to the NSX Security Group created in the previous step:

Then, in Panorama, create the Policy you want to apply to the redirected traffic:

Back on NSX, create a redirection policy, or Partner Security Rule, for the interesting traffic that will be subject to inspection (Network Introspection). In this example we are redirecting inbound HTTP/HTTPS traffic for additional security controls:

Note 1: You will also need to create DFW rules to allow the traffic that will be redirected, as these rules are applied prior to the redirection for outbound traffic (VM >> World) and are applied after redirection for inbound traffic (World >> VM). More details on how these flows move through the Hypervisor can be found on the NSX Design Guide.

Note 2: You may need to use the “ApplyTo” Field in NSX to limit the redirection policy to the specific VMs in question.

Finally, you can use OpenStack Nova to boot Instances (VMs) that satisfy the membership criteria of the appropriate NSX Security Group. It is extremely important that you DO NOT attach a Neutron Security Group to these Instances. We are bypassing self-service security provisioning in OpenStack and delegating all security controls to the Firewall Team.

Note 3: If you are using Horizon (OpenStack GUI), you may need to detach the default Neutron Security Group after you launch your Instance(s).

Note 4: Another approach, not covered in this document, has to do with the manipulation of the policy.json file for Neutron, in order to restrict Security Group changes or additions by anyone other than the Admin. In this case, launching Instances without a Neutron Security Group attachment is not required, as the Neutron Security Group that is used would only be modified by said Admin.

Verify your configuration and security policies.

As we can see, the above approach safely integrates value-add security and visibility services into OpenStack today, and showcases the power of NSX as a platform for Private Cloud deployments based on OpenStack.

Follow these links for the previous two articles in our 3-part blog series:

VMware and Palo Alto Networks will be discussing this and many other interesting topics at VMworld Europe 2016 in Barcelona, Spain. Don’t forget to swing by the Palo Alto Networks booth in the Solutions Exchange if you need more information.

OpenStack doesn’t mandate defaults for compute, network and storage, which frees you to select the best technology. For many VMware customers, the best choice will be vSphere to provide OpenStack Nova compute capabilities.

It is commonly asserted that KVM is the only hypervisor to use in an OpenStack deployment. Yet every significant commercial OpenStack distro supports vSphere. The reasons for this broad support are clear.

Costs for commercial KVM are comparable to vSphere. In addition, vSphere has tremendous added benefits: widely available and knowledgeable staff, vastly simplified operations, and proven lifecycle management that can keep up with OpenStack’s rapid release cadence.

Let’s talk first about cost. Traditional, commercial KVM has a yearly recurring support subscription price. Red Hat OpenStack Platform-Standard 2 sockets can be found online at $11,611/year making the 3 year cost around $34,833[i]. VMware vSphere with Operations Management Enterprise Plus (multiplied by 2 to match Red Hat’s socket pair pricing) for 3 years, plus the $200/CPU/year VMware Integrated OpenStack SnS is $14,863[ii]. Even when a customer uses vCloud Suite Advanced, costs are on par with Red Hat. (Red Hat has often compared prices using VMware’s vCloud Suite Enterprise license to exaggerate cost differences.)

When 451 Research[iii] compared distro costs based on a “basket” of total costs in 2015 they found that commercial distros had a cost that was close to regular virtualization. And if VMware Integrated OpenStack (VIO) is the point of comparison, the costs would likely be even closer. The net-net is that cost turns out not to be a significant differentiator when it comes to commercial KVM compared with vSphere. This brings us to the significant technical and operational benefits vSphere brings to an OpenStack deployment.

In the beginning, it was assumed that OpenStack apps would build in the resiliency that used to be assumed from a vSphere environment, thus allowing vSphere to be removed. As the OpenStack project has matured, capabilities such as VMware vMotion and DRS (Distributed Resource Scheduler) have risen in importance to end users. Regardless of the application the stability and reliability of the underlying infrastructure matters.

There are two sets of reasons to adopt OpenStack on vSphere.

First, you can use VIO to quickly (minutes or hours instead of days or weeks) build a production-grade, operational OpenStack environment with the IT staff you already have, leveraging the battle-tested infrastructure your staff already knows and relies on. No other distro uses a rigorously tested combination of best-in-class compute (vSphere Ent+ for Nova), network (NSX for Neutron), and storage (VSAN for Cinder).

Second, only VMware, a long-time (since 2012), active (consistently a top 10 code contributor) OpenStack community member provides BOTH the best underlying infrastructure components AND the ongoing automation and operational tools needed to successfully manage OpenStack in production.

In many cases, it all adds up to vSphere being the best choice for production OpenStack.