CloudBolt Blog

CloudBolt Software announced today the release of a new product in its portfolio - CloudGoat. For five years, CloudBolt's award-winning, cornerstone product has enabled its customers to achieve hybrid cloud management and self-service provisioning, turning their existing datacenters into a cloud provider, providing IaaS and PaaS. The new product is an exciting foray into the new field of GaaS (Goat as a Service).

Bernard Sanders, CTO of CloudBolt Software emphasized that CloudBolt has been lauded for its ability to manage brownfield (pre-existing) environments to the same level that it can manage greenfield (newly built) environments. This new CloudGoat product offering now allows enterprise IT departments to turn actual green fields into actual brown fields. Just as it has done with its flagship CloudBolt product, the company will be making CloudGoat available for download free of charge (trial licenses cover up to 25 VMs/goats), and expect that this product will also be recognized as the best in its class (though it is unclear at this time what class that is).

"A little piece of advice. You see an Agent, you do what we do. Run. You run your a** off."— Cypher, The Matrix

If you are evaluating Cloud Management Platforms (CMPs), make sure to consider whether the solution requires an agent on the VMs it manages. This is important because your CMP should provide visibility and manageability across all servers, not just the ones it built itself. CMPs that require an agent for full management greatly inhibit your ability to on-board previously-existing (aka “brownfield”) servers. Consider the challenge of getting root/Administrator access to all VMs across all environments and installing software on them. Also, test what happens when people provision VMs outside of the CMP by going straight to vCenter, AWS, etc. Will the tool automatically discover and manage them, or will an administrator need to manually install agents first? Also consider which OS versions the agent is supported on, how quickly new agent versions are released to support new OS versions, and what the upgrade process is for existing agents.

Inclusion of an agent makes a CMP more like a monitoring or configuration management system. It blurs an important line between monitoring and operations. Agent installation and inter-process communications also raise legitimate security concerns. By contrast, a product designed as a manager-of-managers gets all the information it needs through the APIs of each virtualization system, public cloud, and configuration manager. The CMP automatically discovers and manages VMs built with other tools in exactly the same way as servers it builds. All systems are equal.

This was our design philosophy for CloudBolt. We recognized that enterprise datacenters have a plethora of pre-existing interfaces and tools. Rather than supplanting or requiring changes to existing operational standards, we complement and integrate with them. Our approach enables IT architects to choose the best set of tools for each specific function.

I would go so far as to say that if a product uses agents, it is not a CMP. A management ecosystem should be comprised of interchangeable parts and components that respect their boundaries, and integrate with each other over published, documented APIs. Administrators should be able to swap out any particular monitoring (or backup, virtualization, or even CMP) solution for another as requirements change or better versions come along, all without affecting other functional areas. This is central to CloudBolt's approach, and it’s one of the reasons it is consistently ranked highest amongst CMPs.

Today, Cisco announced its intent to acquire CliQr, for a reported $260M. This acquisition validates the importance that Cloud Management Platforms (CMPs) play in enterprise IT shops, forming the foundation of a hybrid cloud strategy.

What prompted this move on CliQr’s part? Perhaps they were feeling the heat from the likes of CloudBolt and decided it was time to retreat to safety. In head-to-head engagements, CloudBolt consistently beats CliQr (and all other solutions) in technical evaluations of hybrid cloud managers / CMPs. This includes evaluations from individual enterprise companies that have done their own product bake-offs (such as GE, Williams-Sonoma, Blackboard, and Waste Management), industry awards such as the Modern Infrastructure Impact Award for best CMP, and a number of analyst and third-party comparisons.

People frequently remark on CloudBolt's unparalleled simplicity, flexibility, and extensibility. They’re also impressed with the breadth and depth of our integration with virtualization technologies, public clouds, configuration managers (such as Puppet and Chef), and various technologies & tools including Infoblox, ServiceNow, Slack, HP OO, Docker, vRealize Orchestrator, and many others.

It's unclear what effect CliQr's acquisition will have on their product roadmap and current customers. Much of the value of CMPs comes from vendor neutrality - their non-partisan ability to integrate with the vast range of IT systems from all vendors. It will be interesting to see if CliQr will maintain this kind of neutrality. Will they, for instance, invest in integration with technologies that are competitive to Cisco's offerings such as VMware's NSX?

In the meantime, CloudBolt will continue to lead the way in hybrid / multi-cloud management and to drive IT transformation forward.

How Cloud Computing is Transforming IT

I’ve spent more than 20 years in IT operations. In that time, computing technology has changed dramatically. However, some things have remained relatively constant. IT operations sits at a nexus of negativity. From a business perspective, operations is overhead and its value is difficult to quantify/justify. From a developer perspective, operational standards are needlessly strict and limiting. From a user/requester perspective, IT is a maddening black box where requests go to die.

It’s no wonder that enterprises are embracing cloud computing with its promise of capacity-on-demand service and pay-as-you-go consumption.

Cloud is a compelling alternative to traditional IT. Customers no longer have to accept ‘no’ for an answer. If they don’t like what IT is telling them, they pull out a credit card and get what they want, when they want it.More to the point, cloud computing is transforming the focus of IT discussions from technology to business. As the enterprise world moves to embrace cloud computing, they follow a logical progression:

Does cloud have any benefits for my business?

How can I use cloud for strategic advantage?

How do I shift to a cloud-first approach?

If One is Good, More are Better

One of the key steps in most enterprises’ implementation of cloud is moving from a single cloud deployment to a multi-cloud strategy. The primary benefit of a multi-cloud approach is the ability to choose the best cloud for each workload. But as organizations work with multiple clouds, they are confronted with a confusing array of tools and management interfaces.

This is where cloud management platforms (CMPs) come in. They serve as a common interface across multiple providers and technology stacks. The CMP market has been around since around 2006 but have remained a niche market until recently. As cloud consumption has taken off in the past 3 – 4 years, though, users have gained a clearer understanding of the benefits and challenges with each cloud service type and provider. Recent expansion of the CMP market has mirrored this growth. There are now many CMP providers, each with different specializations and approaches to multi-cloud management. This is why WhatMatrix is adding a CMP category to aid in the evaluation and selection process. You can find it at www.whatmatrix.com/comparison/Cloud-Management-Platforms.

By way of disclosure, I recently left the world of IT operations management to work for a CMP company. In doing research on other companies the space, I had trouble finding a common basis for comparison. Products are built differently, sold differently, and deployed differently. I’d encountered similar challenges in the past when comparing virtualization platforms. When I came upon WhatMatrix’s virtualization comparison, I found just what I was looking for. It gave me the ability to directly compare multiple products and to drill down on the specific features that mattered most to me. So I contacted the folks at WhatMatrix to ask if they had or planned to add a CMP category. Although the category wasn’t actively being developed, they liked the idea and suggested I join them and contribute.

Since I work for one of the CMP vendors being evaluated, this might appear to be a conflict of interest. If my only interest was to promote my own product, I would simply create one of those ubiquitous charts showing all green checkmarks in my product’s column and all red x’s for each of my competitors. That is decidedly NOT my goal. Rather, I want a fair and objective comparison. In return, I gain a deeper understanding of my own industry. This also helps guide my company’s product direction and development priorities. Additionally, WhatMatrix process are in place to ensure community curation and peer review.

While creating the CMP comparison, we’ve solicited input from all the vendors represented. Nearly all have responded to one degree or another. Still, some of the initial assessments are largely the result of my own research. I trust that this will evolve over time and that each vendor will own and maintain their own evaluation. This will both ensure fairness and help to advance the value of this comparison and of the CMP market.

Evaluation Methodology

When structuring the evaluation criteria, I began with high-level value propositions. What problems do CMPs solve? My list was drawn from discussions with analysts, personal experience with IT challenges, and a survey of solutions from industry players. I sorted the criteria into the following categories:

General: How the product is structured, procured, installed, and managed

Multi-Cloud Support: The ability to manage a variety of clouds and technologies, from virtualization and private cloud to public cloud providers

IT Automation: Features that allow IT processes to be partially or completely automated

User Self-Service: Enablement of end-users to provision systems, environments, and/or application stacks themselves in near real-time based on templates set up and managed by IT

Chargeback/Cost Transparency: Increased visibility into what systems cost and how those cost can be actively managed

Governance & Security: Capabilities to ensure compliance and to safeguard information and operations

DevOps: Features that bridge the divide between developers and IT operations

And so it begins…

This blog marks the launch of the CMP category on WhatMatrix. While I’m very pleased we have reached this milestone, I recognize that this is only the start. The success of our efforts will be judged by the number of products compared and by the degree to which the comparison is seen as fair, relevant, accurate, and current. To that end, I’m reaching out to representatives of CMP companies and to the wider WhatMatrix community to join in the CMP evaluation as category specialists.

TL;DR Like many others, I used to believe Cloud was just a computer somewhere else. I've since come to the realization that Cloud is an application that abstracts and virtualizes the operational details of the servers, software, networks and storage required to deploy modern Internet-based applications. If Enterprise IT is to survive the onslaught of Cloud, they must adopt the same operational efficiencies as AWS and other public cloud providers. This includes exposing IT as a Self-Service Application or Portal. Only then will they be able to stave off Shadow IT and offer their customers the ability to develop, deploy, and manage applications without the day-to-day involvement of IT Operations.

I freely admit it took me a while to come to terms with the term "Cloud". For years I used the infamous Visio cloud shape in my network diagrams. Still, it was difficult to buy into "cloud" as a revolutionary new place where data could be stored centrally and transferred locally over high-speed Internet. "Cloud" became a term that I viewed as spin on the old: mainframe, client/server, web hosting, ASP, MSP, CSP. In short, "cloud" was a new term for the same thing – my "stuff" is stored and running elsewhere. And yes, I admit to getting a kick out of watching Larry Ellison poke fun at Cloud Computing back in 2008. With his usual flair, Ellison pointed out that "Cloud Computing" covers everything we were already doing. In the late 2000’s, everything and anything in the computing industry was described as Cloud, Cloud-ready, or Cloud-enabled.

For years I wholeheartedly agreed with anyone that claimed that “cloud” was just a marketing buzzword. With time, though, my attitude slowly shifted. Cloud isn't necessarily a set of servers that live on a faraway network. A true cloud is a software application, complete with its own UI, API, user management, and a suite of opaque services. Virtualization at the server, network, and storage levels allow users to interact with abstractions that look and feel like physical systems. Cloud architecture is yet another example of Marc Andreessen's declaration that "Software is eating the World", only this time it's enterprise IT being served up for breakfast, lunch, and dinner.

With the right amount of vision and execution, enterprise IT stands to benefit from cloud computing. But it will require them to provide users the ability to utilize resources on their own terms. This is a delicate balancing act:

Too many restrictions from IT will drive users into the arms of public cloud providers while fueling shadow IT.

Running a wide-open environment will come at the expense of security, reliability, and maintainability.

If enterprise IT is going to survive and thrive in the face of public cloud, they need to balance corporate governance needs against user self-service and operational agility that encompasses what the leading public cloud providers are already doing – certainly no easy task.

First this requires embracing the fact that service desks and request tickets are relics of IT past. Users (specifically developers) want fast, easy, and reliable access to IT resources and the freedom to experiment and innovate without the overhead of passing everything through IT operations (ITOps). With the new enterprise Cloud, developers and product teams become responsible for the deployment and management their own applications. ITOps provides the platform for enterprise cloud applications (web-based or otherwise) which manages all the backing networks, systems, and storage. Unless a problem percolates up to the application level, users have little-to-no visibility into what's happening behind the scenes.

As a test of this model, think of AWS. Do you frequently file trouble tickets with AWS? Do you constantly hear about day-to-day operational difficulties at AWS? Do you find yourself waiting until some AWS team returns from lunch to get the servers or services you requested? If your enterprise IT team is embracing what I feel is the true meaning of "Cloud", then asking these same answers of your enterprise cloud should yield the same resounding "NO!"

CloudBolt’s simply powerful cloud management platform has always been available as a virtual appliance. We enable you to manage your virtualization, private cloud, and public cloud environments all in one place. Because our customers work in multi-cloud environments, though, we’ve received multiple requests to run CloudBolt on-demand in the public cloud. We listened, and we’re pleased to announce CloudBolt availability in the Microsoft Azure Marketplace as a pay-as-you-go application.

CloudBolt Free 25 VM Pack is a Bring Your Own License version that’s free for non-production use for up to 25 virtual machines. You pay only for your Azure instance time. Otherwise, it’s free to use, forever. Whether you’re just beginning with multi-cloud management or you’re testing a variety of CMP products, this is a great way to get started with CloudBolt. All you have to do is pick your Azure instance, request a license by return email, and follow the quick installation guide. You’ll be up and running in minutes.

CloudBolt 125 VM Pack is an on-demand version for managing up to 125 virtual machines. You pay only while the instance is running, and usage is billed through your Azure account. The license is built in. You pay a low hourly rate along with your Azure usage. Simply pick your Azure instance and follow the quick installation guide. Before you know it, you’ll have powerful cloud management, IT automation, user self-service, and usage/chargeback reporting at your fingertips.

If you have more than 125 VMs – as most of our customers do – contact us at info@cloudbolt.io. We can easily manage thousands of VMs across a wide range of virtualization and cloud environments from a single CloudBolt instance. And we work with legacy, brownfield environments as well as new, greenfield deployments. All of which makes CloudBolt the central console for management, security, reporting, and control of your entire IT infrastructure.

Cloud computing is all about ease of consumption. By offering CloudBolt in Azure, we’re making it that much simpler to manage multiple clouds from the cloud.

This is the time of year when technical publications solicit prognostications. We received multiple queries here at CloudBolt. In reply, CloudBolt CEO Jon Mittelhauser and Marketing Director Ephraim Baron took turns gazing into the Mirror of Galadriel and reporting what they saw. They offer their technology predictions for the coming year in two separate articles.

Want to get a text message when your VM is provisioned? Want to have a Google doc updated anytime somebody provisions a new VM? Want to be silly and have your stereo play “Die, Die My Darling” every time one of your VMs is deleted? It’s easy with the power of WebHooks and CloudBolt!

CloudBolt has always been able to be extended via our orchestration hooks and custom code. This gives an almost unlimited ability to integrate with external systems. This is (obviously) very powerful but that power comes with inherent complexity and requires some basic programming ability.

One of the relatively new features in CloudBolt is the ability to tie CloudBolt Actions into arbitrary webhooks. This can be used to do all of the above examples and pretty much anything else you could think of… all so easily that even the pointy haired boss (otherwise known as me) can do it!

To prove this, I made a quick video. Here you'll see that in 10 minutes(!), I was able to setup CloudBolt to call my cel phone whenever a VM in my AWS account was deleted - and even tell me what VM was deleted...

To do this, I took advantage of Zapier and its ability to tie various Web apps together. You could use this same capability to tie CLoudBolt into pretty much anything... Twitter, Facebook, HipChat, Slack, Asana, etc.......

So... if even the pointy haired boss can do it, you have no excuse not to!

Please tell us about all the cool systems you've integrated into your CloudBolt ecosystem. We'd love to hear about it!

UPDATE: As of Verison 5.3.1, it will no longer be necessary to check EC2 instance reachability. This functionality has been rolled into the product. This article is still a great example of what it takes to write a CloudBolt plug-in and will be useful in many other scenarios. ~Rick

A common use-case I see frequently is the need to make sure new EC2 instances are up and ready to accept SSH connections before CloudBolt marks the provisioning job as complete. In this article, we’re going to work together to write a CloudBolt plug-in that will add this functionality to our CloudBolt environments. In doing so, I hope you'll not only gain an appreciation for the power of CloudBolt as a cloud automation platform, but you'll also see how easy it is to extend our base feature set using upgrade-safe scripts.

Getting Started

Writing Python code is a relatively painless process that usually starts with a text editor. I use OSX, so I prefer TextMate. If you’re a Windows user, I suggest Sublime Text 2 (http://www.sublimetext.com/2) or Notepad++. Another great option is to use PyCharm for all your CloudBolt plug-in development projects. I plan to expand on this topic in a future article.

Planning Our Attack

Let’s talk briefly about what we want to accomplish with this plug-in: When we provision a VM to EC2 via CloudBolt, we want to wait until that server is finished initializing and ready for SSH access before marking the entire CloudBolt provisioning job as complete. By default CloudBolt marks the job complete once the VM state is set to “OK” by AWS. Unfortunately, this isn’t the full story on the VM's readiness. The “OK” state is set before the VM is initialized and before the user can login via SSH. Imagine your poor users – they just used the awesome CloudBolt platform to spin up a VM, and once their job is “complete”, they get a “Connection Refused” error when they try to connect via SSH – not cool.

To address this issue, we'll extend CloudBolt to wait until our new EC2 instance has passed all EC2 status checks before marking the job as successfully completed. To accomplish this, we’ll trigger an action at the post-provision stage of the “Provision Server” Orchestration Action that will poll EC2 every two seconds to see if our new instance is reachable according to the EC2 status checks. We‘ll implement this action as a CloudBolt plug-in script written in Python.

Starting our Plug-in

Let's start our plug-in with a file called “poll_for_init_complete.py” with the following contents:

defrun(job,logger=None, **kwargs):

return"", "", ""

The CloudBolt platform knows to call this function when it‘s time to execute the plug-in, therefore it's essential that it exists in your plug-in script. Note that the first and required parameter to this function is called job. This implies that we should expect the CloudBolt platform to call this function with the originating provisioning job passed as a job.models.Job object.

Returning a tuple of ("", "", "") is the default way of communicating to the CloudBolt platform that the script was a success.

Let's Get Busy

Let's add a few more lines to our plug-in script to get the server (our new EC2 instance) from the Job object and wait until it's reachable:

import time

from jobs.models import Job

TIMEOUT =600

defrun(job,logger=None, **kwargs):

server =job.server_set.first()

timeout =time.time()+ TIMEOUT

whileTrue:

ifis_reachable(server):

job.set_progress("EC2 instance is reachable.")

break

eliftime.time()> timeout:

job.set_progress("Waited {} seconds. Continuing...".format(TIMEOUT))

break

else:

time.sleep(2)

return"", "", ""

Let's walk through what what we have so far:

server = job.server_set.first() sets the variable called server to the Server object associated with this job. Since we're working with a server provisioning job, it's safe to assume we're only going to have one Server associated with this job, therefore we call first() on our job's server_set property.

We defined a constant called TIMEOUT in our plug-in module and set it to 600. We then use this TIMEOUT at timeout = time.time() + TIMEOUT to set the time at which we should no longer wait for our EC2 instance to initialize. This prevents CloudBolt from waiting indefinitely if for some reason EC2 cannot determine the reachability of our server. Since this is in seconds, we'll stop waiting after a maximum of 10 minutes has passed before marking the job as complete. This should be the exception – not the norm.

We then start an infinite loop that will only stop when either our timeout elapses or we determine that our EC2 instance is reachable with the function is_reachable(server) which we haven't yet defined.

Is it Reachable or Not?

The script above is still missing the implementation of our is_reachable function. Given the server object associated with this job, this function will use the AWS Boto API to determine the reachability status for our new EC2 instance. Note: Boto is the name of the Python API used to access the AWS API.

instance_id = server.ec2serverinfo.instance_idGet the EC2 instance ID associated with our new server being provisioned through CloudBolt. This is a string that looks like i-2423c494 in the EC2 console.

ec2_region = server.ec2serverinfo.ec2_regionGet the AWS region into which our new EC2 instance is being deployed.

A few CloudBolt platform API gymnastics to get the backing Boto API objects without specifying any credentials. Always keep credentials out of your scripts!rh = server.resource_handler.cast()rh.connect_ec2(ec2_region)wc = rh.resource_technology.work_class

return True if status[0].instance_status.details[u'reachability'] == u'passed' else FalseIf the reachability status for our server is “passed”, return True because our new server is now reachable. If not, return False. We use status[0] because our get_all_instance_status function above returns an array. In this case we're only asking for the status of one instance, so we know the array only has one Status object and thus we use status[0].

Going back to our loop you can now see how the is_reachable function is used to keep the loop going if the answer is false:

whileTrue:

ifis_reachable(server):

job.set_progress("EC2 instance is reachable.")

break

eliftime.time()> timeout:

job.set_progress("Waited {} seconds. Continuing...".format(TIMEOUT))

break

else:

time.sleep(2)

If our server is NOT reachable, and our timeout hasn't expired, we wait two seconds and try again.

Putting it All Together

Now that it's ready, let's add it to the appropriate trigger point in CloudBolt.

In your CloudBolt instance, navigate to Admin > Actions > Orchestration Actions and click “Provision Server” on the left tab bar. Find the “Post-Provision” trigger point at the bottom of the page and click the “Add an Action” button.

Select “CloudBolt Plug-in” and in the next dialog, click "Add new cloudbolt plug-in".

Specify a name for our new plug-in (Poll for EC2 Init Complete), select the "Amazon Web Services" resource technology, browse to your script, and click "Create". Selecting the "Amazon Web Services" resource technology ensures this plug-in only runs against AWS resource handlers that you've defined and not others to which this plug-in is not applicable.

Give it a try

Provision a server to one of your AWS-backed CloudBolt environments. Watching the job progress, you'll see that the job is not marked as complete until the server is fully reachable and SSH access is available.

A while back, I was working for a large storage company. We had a marketing campaign called “Journey to the Cloud” where we advised enterprises about cloud computing – as we defined it. For us, the cloud was all about storage. Of course, for server vendors the cloud was all about servers. Ditto for networks, services, or whatever else you were selling. There was a lot of “cloud-washing” going on. I knew we’d reached the Trough of Disillusionment when, as I got up to present to a prospect, they told me “if you have the word ‘cloud’ in your deck, you can leave now.”

Fast-forward five years, and cloud computing appears to have reached the Slope of Enlightenment. By nearly all measures, cloud adoption has increased. Ask any CIO about their cloud strategy, and they’ll give you a well-rehearsed answer about how they’re exploiting cloud to increase agility and drive partnership with the business. Then ask, “How are you enabling user self-service?” Typical responses start with blank stares or visible shudders, followed by “oh, we don’t do that!” They may say “we’re only using private cloud”, or they may mention OpenStack or containers. If so, you should point out “If it isn’t self-service, it isn’t really a cloud.”

Defining Cloud Computing

When looking for a definition of cloud computing, the National Institute of Standards and Technology (NIST) version is widely cited as the authoritative source. NIST lists five “essential characteristics” of cloud computing. The operative word is ‘essential’; not suggested; not nice-to-have. If a service doesn’t have all five, it’s not a cloud. These include:

Broad network access

Rapid elasticity

Measured service

Resource pooling

On-demand self-service

For the last of these, on-demand self-service, the cloud test is simple. If users can request systems or applications and get them right away – without directly involving IT – they are getting on-demand self-service. If they have to submit a ticket and wait for an intermediary to review and fulfill their request, it’s not a cloud.

Working With You or Around You

At this point, you may be told “we don’t offer self-service because our users don’t understand IT. They need our help.” There was a time when that reasoning may have worked. The C-I-‘no’ of the recent past had the power to rule by fiat and ban anything that wasn’t explicitly on the IT approved list. Users had no choice. But times have changed. Now, users can simply create an account with a public cloud service, swipe their credit card, and get what they want, when they want it.

As a result, companies are seeing a marked increase in so-called shadow IT – pockets of information technology that exist and are managed by users rather than by formal IT groups. And while this may cause wailing and gnashing of teeth by everyone from security, to finance, to IT operations, it’s nearly impossible to stop. The genie is out of the bottle.

Rather than trying to prevent or shut down rogue users, IT must take a different approach. They need to ask their users “how can we help you?” rather than “how can we stop you?”

“Be the cloud, Danny”

If you work in IT and want to stay relevant, you need to be as easy to work with as a cloud service provider. Do that, and users won’t look for alternatives. After all, they have their own jobs to do.

So how do you get started? That’s where CloudBolt comes in. We’re a cloud management platform that was designed from the start with the end-user in mind. We enable systems administrators to establish standard configurations and to publish them to their users via an online service catalog. Users get rapid access to capacity; IT maintains control and compliance. Best of all, CloudBolt isn’t restricted to a single cloud vendor’s services and APIs. We work with more than a dozen cloud providers, from private to public, as well as with a wide variety of configuration management and orchestration tools. We even integrate with legacy, brownfield environments giving you a single place for managing existing as well as new deployments.

If simple and powerful cloud management sounds appealing, try it for yourself. Just download the CloudBolt virtual appliance. It’s free to use for lab environments. Deployment and setup are fast and easy. Before you know it, you’ll be providing real cloud services to your users.