To Kernel or Not To Kernel…are we putting services in Kernel, or leaving them out…This has been a hot topic in the Hyper-Converged storage market.

I’d like to take a few minutes to relay some great thoughts that Nutanix CEO Dheeraj Pandey posted on another blog’s comment section a while back. Reading through it now, I firmly believe it needs it’s own proper post. This content is from Dheeraj, with small edits for blog readability.

The whole management argument of integration is being broken apart. Had that been true, Oracle apps would have continued to rule, and people would never have given Salesforce, Workday, ServiceNow, and others a chance. And this has been true for decades. Oracle won the DB war against IBM, even though IBM was a tightly integrated stack, top-to-bottom. After a certain point, even consumers started telling Facebook that their kitchen-sink app is not working, which is why FB started breaking apart that experience into something cleaner, usable, and user-experience-driven.

These are the biggest advantages of running above the kernel:Fault isolation: If storage has a bug, it won’t take compute down with it. If you want to quickly upgrade storage, you don’t have to move VMs around. Converging compute and storage should not create a toxic blob of infrastructure; isolation is critical, even when sharing hardware. That is what made virtualization and ESX such a beautiful paradigm.

Pace of Innovation: User-level code for storage has ruled for the last 2 decades for exactly this reason. It’s more maintainable, its more debuggable, and its faster-paced. Bugs don’t bring entire machines down. Exact reason why GFS, HDFS, OneFS, Oracle RDBMS, MySQL, and so on are built in user space. Moore’s Law has made user-kernel transitions cheap. Zero-copy buffers, epoll, and O_DIRECT IO, etc. makes user-kernel transitions seamless. Similarly, virtual switching and VT-x technologies in hypervisors make hypervisor-VM transitions seamless.

Extensibility and Ecosystem Integration: User-space code makes it more extensible and lends itself to a pluggable architecture. Imagine connecting to AWS S3, Azure, compression library, security key management code, etc. from the kernel. The ecosystem in user-space thrives, and storage should not lag behind.

Migration complexity (backward compatibility): It is extremely difficult to build next-generation distributed systems without using protobufs and HTTP for self-describing data format and RPC services. Imagine migrating 1PB of data if your extents are not self-describing. Imagine upgrading a 64-node cluster if your RPC services are not self-describing. Porting protobufs and HTTP in kernel is a nightmare, given the glibc and other user library dependencies.

Performance Isolation: Converging compute and storage doesn’t mean storage should run amuk with resources. Administrators must be able to bound the CPU, memory, and network resources given to storage. Without a sandbox abstraction, in-kernel code is a toxic blob. Users should be able to grow and shrink storage resources, keeping the rest of application and datacenter needs in mind. Performance profiles of storage could be very different even in a hyperconverged architecture because of application nuances, flash-heavy nodes, storage-heavy nodes, GPU-heavy, and so on.

Security Isolation: The trusted computing base of the hypervisor must be kept lean and mean. Heartbleed and ShellShock are the veritable tips of the iceberg. Kernels have to be trusted, not bloated. See T. Garfinkel, B. Pfaff, J. Chow, M., Rosenblum, and D. Boneh, “Terra: A virtual machine-based platform for trusted computing,” in Proceedings of the 19th ACM Symposium on Operating Systems Principles, pp. 193–206, 2003. Also see P. England, B. Lampson, J. Manferdelli, M. Peinado, B. Willman, “A Trusted Open Platform,” IEEE Computer, pp. 55–62, July 2003.

Storage is just a freakin’ app on the server. If we can run databases and ERP systems in a VM, there’s no reason why storage shouldn’t. And if we’re arguing for running storage inside the kernel, let’s port Oracle and SAP to run inside the hypervisor!

In the end, we’ve to make storage an intelligent service in the datacenter. For too long, it has been a byte-shuttler between the network and the disk. If it needs to be an active system, it needs {fault|performance|security} isolation, speed of innovation, and ecosystem integration.

One more thing: If it can run in a Linux VSA, it will run as a container in Docker as well. It’s future-proof.

NCC stands for Nutanix Cluster Check, and it is the core framework behind the Health construct in Nutanix OS.

This framework is separate from NOS, in that it can be updated async from NOS. Nutanix Engineering releases NCC updates approximately every 4-6 weeks, but can be quicker depending on the criticality of new health checks as they are developed.

There are plenty of blogs out there talking about NCC’s core functionality, and how to most effectively utilize its power, but I wanted to take some time and highlight one new feature, Automatic Email Digests for NCC Results.

Once enabled, NCC will run a full health check of the system according to the value you set, which can be anywhere between 4-24 hours. This enables even more proactive and aggressive health reporting, outside of all of the standard enterprise grade alerting we have today (Pulse Auto Support Call Home, SNMPv3, Syslog, NOS Alerts, etc).

Once configured successfully, the email will contain a summary of the NCC output and includes details
for any checks that return a FAIL status. To see details for all checks, examine any NCC log files
located in /home/nutanix/data/logs.

These alerts are emailed to the users configured in “Alert Email Configuration”, so make sure your sysadmin DL (or users) are configured (good idea either way).

NOTE: NCC 1.3.1 results emailed to Nutanix support do not automatically create support cases, that is the job for traditional break/fix Alerts and Pulse.

Nutanix cluster’s have an AutoSupport mechanism, called Pulse, that can be configured to sent back configuration/health/coredump information on a daily basis. This is used by Nutanix Support SRE’s to do proactive support, analyze common configurations, and so on.

This data can either be sent directly to Nutanix over the web, or be relayed through a customer controlled SMTP server. Many customers have message/attachment rules/restrictions, and sometimes Pulse ASUP’s can get snagged, especially with large clusters, and especially when the “ALL” setting on Pulse is configured, which grabs a very high amount of coredump and diagnostic data.

This data is zipped into a .gz and attached to an email, so even if it is chalked full of data, it is usually pretty small, maybe 6-10 MB daily.I recently ran into an issue where the customer’s email system was configured to drop messages with attachments greater than 100 MB, and also attachments that, when uncompressed, were greater than 100MB.

This filter rule, combined with the ALL level was causing ASUP’s to not be delivered.

To diagnose this, you can see the email data that the cluster is doing by checking out the /home/nutanix/data/email/ folder.

Since the process that generates the emails is a shared service, it is possible for more than one CVM to generate emails over the life of a cluster.

To track the most recent data down, try this command allssh ‘ls -lah /home/nutanix/data/email’ —- This will give you an output like this one from my lab.

Below, you can see that 10.1.222.62 is doing the most recent work, and there are both “.sent” files, where are JSON formatted files that contain all of the email info, except for the attachments.

In the attachments directory, you will find the .gz files that are being generated and, in this case, flagged.

This discovery method can be used to pull down files that were never delivered, and validate their contents to see why an email filter might be tripping them up.

Nutanix Foundation, for those who don’t know, is the tool used to perform Multi-Hypervisor imaging in the field. All Nutanix nodes/blocks are shipped from the factor with KVM pre-loaded, and Foundation is used to optionally change over to VMware ESXi or Hyper-V (or perform a re-image on KVM).

There are some good articles about previous versions of Foundation that cover the basics well.

This article details the steps to upgrade a Nutanix Prism Central 4.0.1 instance to 4.0.1.1 (or future releases). There are a couple of different methods that can be used, all are covered below.

For those not familiar with Prism Central, it is the Nutanix centralized management platform, and is targeted at environments with 2 or more Nutanix 4.0+ clusters. Prism Central is literally the same Prism HTML5 GUI our customers have come to love, but with the ability to manage multiple clusters, at once in a scale-out fashion.

Prism Central gives administrators and operators the ability to see aggregated stats/alerts/info for all connected clusters, and it gives the ability “hop into” any cluster, using seamless pass through authentication. Prism Central was introduced in 4.0.1, and can manage any cluster running NOS 4.0+. Note that Pro or Ultimate Edition licensing is required on a cluster level, but there is no license file that gets applied to Prism Central itself.

NOTE: This article assumes Prism Central 4.0.1 is already deployed, and does not cover the initial install of the OVA. Installation is covered in detail by Baz Raayman’s Blog, as well as in the portal.nutanix.com documentation library.

NOTE: This blog is for informational and educational purposes only. While I have validated this upgrade process several times in my team’s lab, this may not cover your particular implementation situation. For production deployments, it is always a good idea to consult Nutanix Support. Nutanix SRE’s should be the sole source of truth for target code guidance and related upgrade steps. As with literally every enterprise software, DO NOT randomly apply new code to your production environment, without first consulting the proper resources.

There are two primary ways that a system administrator can apply an update to Prism Central.

Upgrade via Auto Update in the GUI

Upgrade via CLI

Method 1: Upgrade via Auto Update (GUI)

In 4.0+, both Prism and Prism Central give administrators the ability to upgrade apply upgrades via the GUI. If your CVM’s have Internet access, they can pull the upgrade binaries directly from the Nutanix Releases API, which means no manual download / transfer process is required. Optionally, you can have the clusters periodically poll the releases API and automatically download new bundles as they are published.

To access to Auto Update feature, click the “Actions” gear in the top right hand corner, then select “Upgrade Prism Central”. This will bring up the “Upgrade Software” context window.

Click “Upgrade Prism Central”

As mentioned previously, if you have Internet access on the CVM, the CVM will poll the releases API and check for a compatible upgrade. If there is an upgrade available, the option(s) will be presented to you, with a big “Download” button to the right of the proposed software release. Optionally, you have the ability to “Enable Automatic Download” via the check box in the lower left hand corner.

Upgrade Software Window

If your CVM’s don’t have Internet access, you will have to download the target code from the Nutanix Support Portal.

Nutanix Releases Download

NOTE: If you have to manually download a NOS release, you will need both the Binary File and the Metadata File. The Binary file, delivered as a tarball, contains all of the actual NOS code. The Metadata file, delivered as a JSON, and is the descriptor for the associated Binary file. This contains, among other things, the expected file size and MD5 hash. These are used together to ensure valid and accurate Binary file delivery.

Contents of Metadata JSON File

To use the Binary and Metadata file, choose the “upload a NOS binary” in the “Upgrade Software” window, select the appropriate files, and then click “Upload Now”. Just like the “Automatic Download” function, this will not apply the upgrade until a cluster admin selects “Install”, after the upload (or download) is complete.

Manual Binary/Metadata Upload

Once the files are uploaded (or downloaded), an “Install” button will appear (not pictured), which will kick off the rolling upgrade. In the case of Prism Central 4.0.1, this will apply the code, and reboot the VM upon success.

After clicking “Install”, you will see a blue circle up in the top left hand corner, which will indicate the progress of the upgrade.

Prism Central: Pre-Upgrade Progress

Prism Central: Upgrade In Progress

After the upgrade is complete, and the VM has been rebooted, you will be kicked back to the Prism Central login screen. After re-authenticating, you can check the “About Nutanix” screen, located under the “User” menu in the top right hand corner.

About Screen: Upgrade Success!

Method 2: Upgrade via CLI

If you are glutton for hard work, or just enjoy the peace and quiet of a black and white command prompt, you can also upgrade Prism Central via CLI, utilizing both SSH and SCP.

Upload the Binary and Metadata files (described in the previous method) to /home/nutanix using your favorite SCP client (I like WinSCP).

Second, log into the Prism Central VM via SSH, and execute the following commands:

This will kick off the upgrade process, just like in the GUI. You should immediately see the effect in the GUI, which is the blue progress circle. In the CLI, you will see console output of whats actually going on.

CLI Upgrade Console Output

After that console output ceases, the upgrade will continue to process in the background. To check status, use the “upgrade_status” command.

upgrade_status command output

As with the GUI method, this will take a few minutes, then reboot the VM upon success.

If you want to check the version/upgrade history, run “cat /home/nutanix/config/upgrade.history”.