Cloudstack... Seems neat. Anyone actually get it to work?

I've spent the last 3 days banging my head against this thing to try and make their latest release work properly. It's my first foray into open source virtualization stuff but the concepts are pretty clear in my head.

I started off with 2 servers, one for manager and one as a KVM host server. Unfortunately, no matter what I have done to try and follow their install docs and administration, I can get close but no cigar. The closest I got was the ability to start the 'console VM' that is part of the system install that lets you see the virtual console on any of the VMs. I got that one started successfully once. There is a second system VM (secondary storage) that appears to have a role of managing the system's template files and ISO files for deploying any additional, actual user VMs. That one never starts. Most of the time, the initial setup fails at the last step 'adding the host'.

Most of my frustration stems from the obtuse interface which has obviously seen some of the worst behavior of open source software (schizo interface designs and horribly inconsistent user/admin prompts and help dialogs). Basically, you have to know how to set the system up before you can set it up and the install guides cover about 10,000 variations without providing much guidance on which is preferred or more compatible.

I've read through both the basic and advanced install guides and come away more confused than before, really. It's not clear what is to be done in advance of starting the install and what is done by the initial setup or what is done each time you add a host server (if I ever got far enough to add a second one).

So has anyone had any good experiences with this stuff? Their forums seem to be a more or less typical pile of unanswered questions and complaints or questions that match my issues but are inevitably answered with 'never mind, I figured it out' but no details on the resolution.

We are supposed to deploy it once we have shared storage and subnets that we can dedicate to it. Frankly it looks like it may be too complex for our needs. If we were starting from scratch to build out an entire virtual infrastructure it might work, but we will be running a hybrid config with legacy physical servers for quite awhile.

It's weird, it is like it assumes you know nothing about servers and networking at some parts of the setup and tries to do everything for you (with little to no manual intervention possible) and at other points it is totally open to you typing in whatever and possibly destroying the whole thing. Sometimes it catches a gross config error but doesn't provide any real feedback on how to fix it or the appropriate values to use.

Out of frustration and a passive-aggressive desire to punch cloudstack in the collective jimmies, I tried proxmox. 15 minutes later I have a working KVM management system with a single host server and I am running a windows 2008 r2 sp1 install.

So yeah, it's not just my fault (though there is probably something I am doing wrong with cloudstack).

I tried posting on their forums at cloudstack.org and after 30+ views, not a single response.

There are a bunch of use made guides that cover a lot of other topics that can cause problems during install (like if you want alternate storage platforms, bare metal server setup, etc.) but nothing that just goes right into the details of 'when you setup the system, you should put the following values in the following fields for the following reasons', which is what I think I really need.

Well, I finally got a bit of clarification on the cloudstack forums. The basic install option is designed for an all-inclusive 'flat' network of a single subnet with a single network interface on the constituent servers and a gateway device of your choosing (router/firewall).

The advanced option suddenly involves not just the ability to use multiple subnets or interfaces but requires the use of VLANs apparently. Now, while I intend to use a config like that for a production environment, their management interface and documentation of how such a design would work is woefully inadequate. Basically it is non-existent.

So, I went back to the drawing board, did a fresh Centos 6.2 install on 2 servers with their eth0 ports in the same layer2/3 network and started with a basic install.

I learned the following items that helped me get thing going:

- Put static hostname entries for the servers and their IPs in your /etc/hosts file even if you have valid DNS (forward and reverse). I know, it should work without this, but it doesn't for me.

- On the host server, create a cloudbr0 bridge interface and bind your eth0 interface to it (move your IP config to it as well) before installing anything of significance.

- Make sure sshd is configured to PermitRootLogin unless you want to create a highly privileged user for the servers to talk to each other and do all their work.

- Don't just set the password for the mysql server and assume the setup script will do the rest. Depending on how hostnames and IPs resolve, you need to GRANT ALL PRIVILEGES (WITH GRANT OPTION) on *.* to the database user and password you want to use, and you need to do it for any IP/Hostname combo that might end up getting used during the system setup or later.

- Make sure your BIOS has the VTd (virtualization) feature enabled.

- When you do the basic install setup and it fails at some point, your options are to: - A. If it is not very far into the setup - Wipe your machine and start over (maybe both to be safe) - B. If it is not very far into the setup - Try to surgery out the stuff in the database that might be wrong or the various agent and server config files that might be wrong - C. If it got all the way to adding the host and failed - Wipe your machine and start over (maybe both to be safe) - D. If it got all the way to adding the host and failed - Let the setup error out, try to put the host in Maintenance Mode and then remove it. Then reboot both servers and try to add the host again. Pray it works

- Adding the system VMs that manage DHCP and shared storage is slow unless you have very nice hardware.

- Watch the logs in /var/log/cloud* but don't expect them to be very helpful. They are cryptic and sometimes maddeningly generic.

- It appears to be running some sort of firewall by default that blocks all ingress traffic to the VMs. Silly. If I want that I will either turn it on manually or do it on the hosts.

Anyway, I have a VM running finally. So I can start to get an idea of how this thing works and whether it is more attractive than an OpenStack setup.

Well, after all the hair pulling and pain to get one VM up an running, the system is kind of cool. Unfortunately my second VM deployment was so slow and difficult I gave up. Before trashing the install, I poked around all the different options in the interface. There are a lot of things to do. Unfortunately most of them are hard to find, hidden under strange tabs, sliding panels and layered menus.

In the end, it is somewhat needlessly complicated from what I can see. A simpler, more unified interface design with increased reliability and ease of setup and management would go a long way.

We have more or less decided to look much harder at openstack now. Cloudstack seems to be an unfortunate middle ground that sacrifices the simplicity and ease of use/setup of Proxmox for more features but incurs a high cost of over-complication, an unwieldy interface that obscures most of the actual stuff you need to make it work and leaves out a lot of critical information. The lack of decent documentation just puts the whole thing on the scrapheap.

I can see why they went to an apache license. Essentially at this point, if you are not willing to pay the cloudstack/citrix guys to setup and support the thing for you, cloudstack is more or less a dead duck. I think with the apache license they want to attract enough attention from the open source crowd and the hardworking tinkerers that eventually things will just get done for them.

Given the approaching 'big step up' from Microsoft with Server 2012 and its heavy focus on virtualization, command line interface and streamlining, I don't think cloudstack stands much of a chance of sticking around very long.

I am actually using CloudStack at some scale. (small, 5x DL 360 G7 and Isilon storage) I agree that initially it can be a little difficult to install but I must say the documentation and support through their IRC channel is actually pretty good. I have found it pretty powerful, we have 80+ largish VMs (up to 4vCPU/16GB RAM offerings) hosted for our development and professional services groups with multiple Windows and Linux based templates and it's been rock solid.

Honestly you can't expect ANY Cloud orchestration tool to be easy, it is inherently complex with clusters of hypervisors, storage pools and network isolation.

I evaluated OpenStack, SCVMM2012, Eucalyptus and VMWare vCloud and this by far was the most mature, "ready to do some actual work" solution out there.

I am a fixture on their IRC channel if you are looking for some help. #cloudstack@freenode

So CloudStack supports multiple hypervisors in a cluster configuration.(XS,XCP,KVM,OVM,ESXi) I am using mostly XenServer 5.6 SP2 and I have used XCP (Xen Cloud Platform, think it it as the Fedora of the Xen world. XenServer would be the Red Hat)

I have done a little testing of KVM but I found it a little too fickle for my tastes. CloudStack for instance requires a specific version of libvirt that they have packaged when you install KVM hosts. In any event I personally find XenServer/XCP easier to use.

As for my deployment, I am running 2.2.14 as there isn't an upgrade path to 3.0.1 just yet. Networking is easily the most complex part of the setup. I am using advanced networking. Basically, every user account has their own VLAN ID for Guest VM instance isolation. A virtual router sits in between this guest network and my public network. The user must create port forward and/or firewall rules through their UI or API to allow inbound network access (kinda like EC2). This of course requires my hypervisors hosts to have a dedicated NIC for guest traffic and a private switch(s) allowing for tagged VLAN traffic.

You could also do basic networking, Basic networking does isolation via ingress/egress firewall rules (just like EC2) you would have a seperate subnet or range for your guest traffic. Basic networking doesn't have all the bells and whistles of advanced but it scales better if you are doing a big deployment. I will say that in version 3.0.1 that basic is almost feature complete as advanced.