2013-06-19

That was the call I got today. "All I did was add a VMkernel interface and my host lost connection to vCenter".

On went my troubleshooting hat.

First the environment (simplified)

The physical interfaces on which the VMkernel interfaces reside were trunked with multiple VLANs. In this case VLAN(4) and VLAN(49).

vmk0 was used for ESXi management - with a default gateway of x.x.4.254

When the user added vmk1 - the host would become disconnected, he removed vmk1 - the host reconnected to the vCenter.

While the host was disconnected, we tried to ping the vmk0 interface - replies were fine.

While the host was disconnected - we tried to connect to the host directly with SSH and the vSphere client - both worked.

While the host was disconnected - we tried to ping the vCenter server with its IP - there was no response.

While the host was disconnected - we tried to ping the external network - replies were fine.

We then looked at the settings on vmk1 - and I noticed that the user had not set the VLAN49 tag on the VMkernel interface. Obviously this was not set correctly, and by adding VLAN(49) to vmk1 - everything worked correctly. The Host reconnected to vCenter.

So the problem was solved - VLAN(49) was missing on vmk1.

I was puzzled and tried to understand why this misconfiguration would cause the host to disconnect from the vCenter - then I realized why, and therefore the reason for this post.

When configuring a VMkernel interface, a new entry is added to the routing table. There will only be one default gateway - and that will the one defined on the Management interface. The additional VMkernel interfaces will not have a gateway defined.

This was the printout of the esxcfg-route -l from the Host.

Just to explain the output in plain English.

Anything on Network x.x.4.0 on that subnet will go out through vmk0.Anything on Network x.x.6.0 on that subnet will go out through vmk2.Anything on Network x.x.49.0 on that subnet will go out through vmk1.Everything that does not match the above - will go out the Default Gateway x.x.4.254 through vmk0.

So the user had configured vmk1 on x.x.49.0. That meant any traffic trying to go out to the vCenter server - would go out through vmk1 - it was on the same subnet.

But… the user had not assigned the appropriate VLAN(49) tag to the interface - which meant that the interface would send out packets onto the network but without the correct VLAN tag on the packets, and therefore the Host could not communicate with the vCenter.

That is a very legitimate question!! Giving a user all rights with no password can become an issue. But in this case since it is used (most probably) for the purpose of the upgrade of the VCSA from version to version - then it might OK (or not… ).

Which leads straight into the next subject - what users (and what are their rights) exist on the vCenter Server Appliance?

Let's have a quick look at what users there are on the VCSA (screenshot is all the local users on my VCSA that are both not locked or disabled)

So firstly let me say that is a hell of a lot of users in the server that in essence is one of (if not the most) important part of your Virtual infrastructure.

You might say (and rightfully so) that even if the users are defined - that does not mean that they can actually log into the system - for that they would need some kind of shell access. So I checked which users actually have shell access. That would be users who do not have /bin/false or /sbin/nologin in their profile. This is what I got

Let's concentrate on some of the users in the list.

lp

Access to printer hardware; enables the user to manage print jobs.

ftp

FTP user

man

Used for man

games

Access to some game software

news

Used for news application

uucp

Serial and USB devices such as modems, handhelds, RS-232/serial ports.

For someone that values security - all our Linux boxes are hardened (as they should be), so the first thing we do is run the following:

(Just to clarify - I do not advise running any of the above commands on your vCenter Server without proper testing and approval from VMware and doing so might void your support. Be warned!!)

If you were to ask me - most of the users listed above have absolutely no business being on a production box… especially not on my vCenter!!!!

In VMware's defense I should say that I checked which users were available on other VMware appliances - such as the vCOPS appliances and the new Log Insight appliance - and most of these users were not present on either of them. Perhaps this is the way going forward.

vCenter has all the keys to the kingdom - and VMware must make the utmost effort to make sure that no possibility of exploit can used by leaving silly holes and possible security vulnerabilities open in the underlying operating system.

The same way that a ESXi host is locked down - there is no reason why the vCenter server should users like games on them.

I have heard from within VMware that Paul Strong was not appointed as the new CTO (he is the CTO for Global Field- but not THE CTO), so I do not know where The Register got their information from.

Which now brings me to another and more important question. How long can VMware go on without filling Stephen's shoes? Stephen left his position as the CTO on January 15th, 2013.

Shortly thereafter (related or not I do not know - I am not a financial buff) the VMW stock took a beating, dropping almost 20 points, today the stock it still at around the same level (after some ups and downs).

Stephen was (and still is) highly respected for his leadership qualities, his vision and his part in bringing VMware to the place they are today. I do not think there were many who were not shocked about the announcement - it was not something that many predicted. The community - especially the active virtual community hold him in high regard.

So the obvious question is what is taking so long to replace him? I do say that his replacement will have to live up to high expectations, from the shareholders, from the community and also from the customers. His spot is not an easy one to fill. I personally do not know enough about the "visionaries" within VMware who can replace him, or perhaps it will be someone from within the "mothership" (gotta love speculation).

I do think that VMware have to fill this void - the sooner the better. It has been 5 months. It is not good for business, it is not good for your stock.

It will restore peace to the the world, calm to the schizophrenics and ..

New Third-Party Plug-insRed Hat is already collaborating with several industry leaders to integrate their solutions with Red Hat Enterprise Virtualization via the new plug-in, including high availability and disaster recovery solutions from NetApp (with a VSC), Symantec (HA), and Insight Control from HP

Mirantis, and Red Hat today announced that the two companies will collaborate to optimize Mirantis’ Fuel tools for deployment of Red Hat OpenStack, and deliver OpenStack implementation and integration services to joint customers.

New solutions include extension to Red Hat Enterprise Linux product family and new offering to enable customers on their journey from datacenter virtualization to Infrastructure-as-a-Service. Red Hat today announced two new product offerings with one vision of delivering an Open Hybrid Cloud. The new offerings include Red Hat Enterprise Linux OpenStack Platform, a solution that serves as the foundation for advanced cloud users who are seeking to build an OpenStack-powered cloud, and Red Hat Cloud Infrastructure, a comprehensive offering designed to support organizations on their journey from traditional datacenter virtualization to OpenStack-powered clouds.

2013-06-11

VMware vCenter Log Insight is the new solution of VMware for log management and analytics for dynamic hybrid cloud environments. It delivers superior technology for automated log management through log analytics, aggregation, and search to extend the leadership of VMware in analytics to log data. Log Insight can analyze vast amounts of unstructured machine generated data and enable interactive, real-time search and analytics through an easy to use interface providing superb time to value. It analyzes log data of all types and from all devices, enabling deep, enterprise-wide visibility. With a focus on integrated cloud operations management, and an analytics driven approach, Log Insight provides the operational intelligence needed to proactively enable service levels and operational efficiency in dynamic hybrid cloud environments.

Think of it as something similar to Splunk but different - it is specifically vSphere Centric (at least at the moment), built by VMware people (as a result of the Log Insight acquisition from August 2012) and it integrates with vCOPs (which is a great plus)

VMware vCenter Log Insight is available for purchase as a standalone product. It has a simple pricing model, with one flat rate for any server, virtual machine or vSphere host from which you collect logs.

VMware vCenter Log Insight is licensed on a per operating system instance (OSI) basis, which is defined as any server, virtual or physical, with an IP address that generates logs, including network devices and storage arrays.

With Log Insight, you can analyze an unlimited amount of log data per OSI. The advantage of this is a simple and predictable pricing model that is based on the size of the infrastructure; it does not force you to buy additional licenses to cover the worst-case scenario and pay more for increased log volumes.

Given that systems and devices can generate huge amounts of log data during peak times, or while monitoring and troubleshooting for various IT issues, this is an important distinction.

Detailed pricing information will be announced when vCenter Log Insight is ready to ship in Q3 2013.

Just as a side note… Version 1.0 should not have a version number of 0.9.1 - that does not make sense…

2013-06-04

Last week, the 581 people that were awarded the vExpert title for the year 2013. It is large list of people who are active in the community, that share knowledge, that lead VMUG's and all other kinds of evangelizing for VMware and the community in general.

As we all know there are a number of "perks" that come with being a vExpert, but mostly it is an honor.

An honor to be part of an amazing group of people

An honor to serve the community

An honor that people acknowledge your contributions

I have traditionally created a Twitter list of the vExperts each year, and this in not different.

I actually created two of them due to the fact that Twitter limited the number of members of a single list to 500 members (and of course they changed it on Thursday - after the lists were populated)

So there will be only a single vExpert 2013 list which you can find here.

Thank you for the honor and here is looking forward to a wonderful and exciting year..