My Technology Adventure

Category Archives: English

Just a quick post. When virtual machine cannot get DHCP IP address the first thing you want to check is firewall. Whatever Windows firewall or physical firewall. You should make sure UDP port 67 and 68 are not blocked. Otherwise you will see the virtual machine gets 169.x.x.x IP address only.

The two ports is required for DHCP client to query IP addresses. The methodology is introduced in RFC document.

DHCP uses UDP as its transport protocol. DHCP messages from a client
to a server are sent to the ‘DHCP server’ port (67), and DHCP
messages from a server to a client are sent to the ‘DHCP client’ port
(68). A server with multiple network address (e.g., a multi-homed
host) MAY use any of its network addresses in outgoing DHCP messages.

You may see ‘Adobe Flash Player Out of Date’ on Chrome when you open vSphere Web Client. Click the text Chrome will update Flash Player automatically. But in some cases it doesn’t work due to maybe your Chrome is controlled by company policy or internet problem to Adobe.com. I found an article to show how to offline fix the issue. You can download Flash Player for Opera and Chromium-based browsers – PPAPI in official Adobe KB article.

You may also want to check out my other articles about Flash issue on browsers.

Slight network latency may cause application problem on sensitive virtual machines. Even the network responding time is just 3 or 7 ms. There is a way to improve the stability of responding latency – Enable RSS on NIC.

Network traffic is handled by single CPU core when RSS is disabled. Enable it will distribute the workload to 4 cores by default. You can increase CPU for RSS by change registry.

If your company implemented firewall and blocked public NTP server, you may see installation of vRealize Operation Manager pending on ./install.sh on console. That’s because the installer tries to negotiate with NTP server http://www.iana.org. The firewall blocked the traffic.

VMware TAM Manager Shan told me there are two options on firewall to block traffic: REJECT and DROP. REJECT means firewall responding to the request and let source device knows it’s rejected. DROP means firewall immediately ignores the request and no responding to source device. Looks like there is a bug in vROPs code that it hung if NTP request gets drop and no responding.

The workaround is create a port group without physical uplinks and install vRealize Operation Manager. Then move it to proper network after installation is completed. You can configure correct IP addresses when import the OVF file so later on you just need simply move the network.

Just noticed a gap here. If your ESXi host is 5.5 Update 2 or earlier version, you are not able to choose ‘Windows Server 2016’ as operating system when you create virtual machine. It’s addressed in KB https://kb.vmware.com/kb/2132789.

The problem was flash menu appears when I right click anything in vSphere Web Client in Chrome. I have two computers that both has Chrome installed but one has issue, other one works fine. I compared version of Chrome, noticed working one was 55.x, problematic one was 49.x. The issue gone after upgraded to 57.x.

After dig into that problem, looks like Google fixed the problem on version 54.0.2840 that there was a bug related to right click. Check out release notes here.

I own Raspberry Pi 1 and 2 both. I used to do some small projects and learn Python on its. It’s great computer to learn scripting and Linux. But not something can be used in daily based.

I was excited when I hear Raspberry Pi foundation released Pi 3. Thanks foundation brought our world a powerful, cheap and flexible computer that can replace my laptop. In general, I think Pi 3 can be used for daily work in GUI interface. It comes with embedded WiFi, Bluetooth and quad-core 1.2GHz CPU! It frees up the 4 USB ports so I can use it for other purpose.

I have a laptop that motherboard was dead, screen still works. So I bought driver board for the LED screen. Connected Pi 3 to the driver board by HDMI cable. The official Pi 2 case is compatible with Pi 3 except power LED in the other side. I use the latest OS “RASPBIAN JESSIE WITH PIXEL“. I have to say the graphical interface is awesome! It’s more like a modern, real computer. I use default browser Chromium which is similar with Chrome PC version. I also installed “Google Input Tool” extension on Chromium so I can input Chinese. Since most of my work on websites so I don’t have to install input tool on OS. And I saw some interface problem when use Chinese input tool for Linux. I also tried bluetooth headset, not working well. So I turned to USB headset, it works perfectly.

I opened 10 more tabs in Chromium just like how do I use my PC. I also ran a online website to streaming music to USB headset. Wrote blog, brows photos, edit files…etc. I didn’t see any performance issue so far. I guess my high performance TF card may helped. I bought Samsung 64GB UHS-1 Class10 for my Pi 3.

Just a quick update that VDP will discontinued in future release of vSphere. It doesn`t impact to existing customers till 2022. VMware offers free migration to Dell EMC Avamar Virtual Edition in limited time if customers want to move to other backup solution. Customers can also use other 3rd party backup solutions to integrate with vCenter Server via storage API.

Just noticed a issue that nothing reported in ‘Hardware Status‘ tab of ESXi hosts in vSphere Web Client. KB 2112847 gives a solution but not works for me. The feature can be used to monitor hardware failures. I figured out a way to workaround it. You just need to login by Administrator account and click ‘Update‘ button under ‘Monitor‘ – ‘Hardware Status‘ for each ESXi host. You will get the status after few minutes.

Virtual appliance is future of how VMware delivery their product to customers. It’s pain to migrate from vCenter Server Windows version to virtual appliance. The only way was build up new virtual appliance and move everything out of Windows vCenter Server. The challenge is you lost data if you have integrated vCenter Server with other VMware products, or using DVS.

VMware released vCenter Server Migration Tool after VMworld 2016. It gave me confidence to give it a try. I assume vCenter Server is embedded SSO. I did the migration 2 or 3 times, following is summary of my experience. The migration tool only support migrate vCenter Server 5.5 windows edition to vCenter Server 6.0 U2.

Prerequisites

vCenter Server is more like core services today since lot of 3rd party software call vCenter API to interactive with VMs. You may have some products integrated with vCenter Server already. Please upgrade to vCenter 6 compatible version before migration.

I suggest create a local account on source vCenter Server if your server is domain member. You can login back source vCenter by local account in case migration failed.

A helper VM is required to run migration image. Please make sure you have a free Windows VM be ready to mount migration image.

SQL database is exported to source vCenter Server if you want migrate performance and event data. So you need to make sure enough space on C: drive on source vCenter. The free space should be much bigger than vCenter database size.

Of course you need a vCenter Server 6 license key since old key doesn’t support the version.

Some cases show migration process stopped during export SQL database. That’s because memory of source vCenter is too small. Please make sure RAM of source vCenter should be equal or greater than destination vCenter Server.

The other tricky is database table. You may see migration processes is completed but destination vCenter Server doesn’t come up, and no data actually imported. That’s because ‘checksum‘ column existing in table [dbo].[VMO_ResourceElementContent] in vCenter DB. You can run following SQL query to remove it before migration.

alter table dbo.VMO_ResourceElementContent drop column checksum;

Procedure

The items above can be done anytime before the migration window. Following steps should be token during migration.

You need to disable firewall and anti-virus software on old vCenter to avoid communication issue between Migration Assistant and new vCenter Server.

To avoid any unstable, resource contention, or potential network connectivity lost issue, I suggest temporarily disable DRS and HA on source and destination cluster if they are virtual machine.

I wrote an article to introducing how to integrate PowerCLI with PowerShell and PowerShell ISE. VMware just released PowerCLI 6.5 R1, it includes lot of new features and modules. And somehow my way doesn’t work. Following is new way to integrate PowerCLI 6.5 with PowerShell and PowerShell ISE in Windows 10.

PowerShell and PowerShell ISE both have it own $profile. So we need to do two times.

Before we start

Please make sure your PowerShell execution policy is not restricted. You can get the setting by run following command:

Get-ExecutionPolicy

PowerShell Integration

Open PowerShell window. Run following command to confirm profile file is not existing.

Test-Path $profile

If return is ‘False’, go to step 2.
If return is ‘True’, Backup the file and go to step 3.

New B200 M4 blades can running on Intel v4 processors. You may see discovery issue if your UCSM firmware version lower than 2.2.7c. I hit that problem few days ago when I install a new M4 blade. The FSM hung on 58% a real long time and failed eventually.

If you have VMware Workstation and VMware Remote Console both installed on Windows 10 you may see vCenter virtual machine console in vSphere Web Client is opened by VMware Workstation instead of VMware Remote Console. If you see same case your VMware Remote Console may be hijacked. I call it “hijack” since there is no easy way to revert it back.

I got a request few weeks ago. Reporting virtual machines that powered off more than 3 days. I didn’t find out-of-box report in vROps can do that. I had to create customized one. Following is how I did.

If you are multi-language IT Pro you may want to show articles in multiple languages just like me. I use Polylang in my blog to switch English and Chinese version. I used to add “Language Switcher” tag in my articles. It’s manual process that I had to remember the tag in somewhere, copy and past it to articles when I was writing. It’s frustration to repeat it in every article. Another problem was I have to add “Read More” tag in each article before I add “Language Switcher” tag. That’s because each post shows a switcher in home page if I don’t do that. It confuses Polylang which URL each article should goes. I want to find a perfect way to fix this problem! Here we go…

There are several ways to reporting snapshots. You can use PowerCLI, VRTools, or even vSphere Client itself. Today I will show you how to reporting by vRealize Operations Manager (vROps) 6.x. The benefit of vROps reports is you can schedule it by sending email with PDF & CSV reports. I found a post discuss how to reports by vROps. But it requests modify policy, it may impacts global calculation. What if your teams request different criteria for reports?

I want to try make a remote control car by Raspberry Pi. The first question is how to power a portable Raspberry Pi? There are lot of solutions in internet. I want to find a simple, cheap and long battery life solution. Some articles introduced power bank but it overs $100 and only provides 5v output. I need some thing can provide not only 5v power for Raspberry Pi, but also more than 7v for L298N.

Cisco UCS blade system is the best blade system I used so far. Whatever the hardware, software or support is perfect. I recommend leverage the system for primary system of virtualization. UCS blade system architecture is different with HP. I feel it more likes a network system. Fabric Interconnect (FI) modules exchange data between uplinks and internal components. IOMs on each chassis controls data routing. Architecture is complicate, but it’s powerful to manage large datacenter. Talking about large datacenter, you may have hundred chassis or blades. Data goes through FIs, IOMs and blades, you could see issues on any layer. It’s hard to find out where exactly the problem is. UCS Manager provides statistics for ports just like how Cisco does on network switches. You can show statistics of a particular port. But it doesn’t tell you when and which layer it happened. I tested Cisco UCS adapter for vRealize Operation Manager before I reviewed NetApp adapter for vRealize Operation Manager. It’s developed by same company Blue Medora. I’d like to introduce few of this product, it’s just my personal review.

Today I got a strange problem about share folder. Some virtual machines cannot access network share path. It gave me Unspecified Error 0x80004005 when I opened a share folder on explorer. It gave me The network path was not found 0x80070035 when I opened same share folder by clicking Start – Run.

vRealize Operation Manager 6 (aka vROps) is new generation of vCenter Operation Manager. I started to use vCenter Operation Manager since version 1.0. I like the idea of self-learning and dynamic threshold. But the product only monitors virtualization layer. It would be perfect if it’s able to monitor under layer storage. In large vSphere environment, virtual machines share IO capacity of datastores. If few virtual machines running high disk IO it may lead to other virtual machines get performance degrading in same storage. Think about you have 100 datastores come from a NetApp filer, and 300 virtual machines running on its. One user says their virtual machine is slow but no workload from applications end. It hard to say where the latency comes from because multiple virtual machines may share same datastore, multiple LUNs share same aggregate, and maybe same physical disks. vCenter Operation Manager provided NetApp Adapter for 5.x few years ago. But the problem was it’s too hard to associate storage objects with vSphere datastore objects.

vRealize Automation 7 (vRA 7) has lot of enhancements and changes compare with vRA 6. There are plenty of introductions available in internet. The initial configuration is different with vRA 6. I’m going to share my experience. You can easily build up LAB or POC by following this post.

You may see that error message “To view this page ensure that Adobe Flash Player version 11.5.0 or greater is installed.” when you open vSphere Web Client 6.0 on IE 11 on Windows 8.1. The login fields still visible, but the page go to blank after you login.

Best practices to manage enterprise Active Directory is organizing servers by particular properties. For example, servers maybe put into different OU by role, business group or function…etc. Following is a vRO workflow sample to automate provisioning computers in proper OUs according to user choice in vRA Service Catalog. I’ll just give brief of each step in this article, so please make sure you understand both products before read this post.

When you put virtual machine to particular OU, you may refer to virtual machine properties, such as ‘server role’, ‘server group’ or ‘user group’…etc. It’s easy to set a drop-list in blueprint of vRealize Automation Center (vRA) to let users choose this kind of properties but hard to create a computer account in corresponded OU location in vRO. That’s because vRA passes most of values to vRO as strings, Active Directory workflows in vRO do not provide a way to convert string to OU.

One day, my vCenter Server suddenly lost search. It popped me “Unable to connect to web services to execute query. Verify that the ‘VMware VirtualCenter Management Webservices’ service is running on https://vCenter_Server_FQDN:10443” when I did object search on vSphere Client. Few hours later people starting complaint they got error on vSphere Web Client, it show “Client is not authenticated to VMware Inventory Service – https://Inventory_Service_FQDN:10443“.

Today I created few super metrics on vRealize Operation Manager 6.0 to calculate throughput of physical links on ESXi host. The super metrics just present to part of the selected hosts. I guess it’s some kind of minor bug. A reboot of vROps vApp can works around it. Just heads up.

I don’t know why VMware doesn’t allow hidden default dashboards in VMware design vRealize Operation Manager (vROps). They also states no solution in current version. I searched internet, only thing I found was a community post that someone wants to delete the dashboards, but no proper answer.

First of all, this article is nothing related to PowerCLI. You probably know how to set Path Selection Policy (PSP) by vSphere Client, but how you can set up 100 LUNs manually? We have some script can make your life easy.

You almost can do everything as long as vRealize Automation Center (aka vRA) and vRealize Orchestrator (aka vRO) are integrated. I think that’s the hard part if you are newbie like me. After reading lot of articles, I learned how it works. Following is my experience, please let me know if you see anything wrong.

Microsoft just released technical preview 3 of Windows Server 2016, it’s catching up VMware on SDDC. I can see a lot of enhancement in the new version. A stable hypervisor is prerequisite of SDDC but it’s weakness of Microsoft. You have to patch and reboot frequently, some organizations even have regular reboot schedules. Microsoft introduced core mode on Windows Server 2008, it much enhanced on Windows Server 2012 R2. But Windows Server 2012 R2 aims to SBM. I didn’t think SBM organizations really need that if you compare operation complexity of core mode with GUI.

ESXi 5.5 Update 2 is stable version, but I got PSOD on one UCS blade few days ago. It scared me since there was a big bug when I upgraded ESXi from 5.1 to 5.5 Update 1 last year(See detail ESXi 5.5 and Emulex OneConnect 10Gb NIC), it lead to dozen virtual machines crashed over and over again.I bet I’m gonna to die if it happens again. :-)

Someone setup a non-secure wifi around my apartment, I never connected it till yesterday since I worried it’s may be a honeypot. I had some me time yesterday night, so I setup a virtual machine to connect the wifi.

I wrote a post about how to integrate PowerCLI with PowerShell manually. I rebuilt my computer few days ago, need to integrate PowerCLI again. I used to scripting by PowerGUI, but something always lead to PowerGUI lost menu, it frustrated me a long time. I cannot figured out what’s the root cause. So I wondered is it possible use PowerShell ISE instead of PowerGUI?

I just heared Transparent Page Sharing (TPS) is disabled by default in latest ESXi 5.5 patch. You may concern about that if your IT budget is tight since it means you need more memory for heavy virtual machines.

It’s been a long time sine last post. I was out of internet due to health issue. Just got recovered and backed to normal work. I have to publish my article by English then translate it to Chinese later since I lost lot of me time after my baby born, but more fun. hopefully it not impact to Google search. :-)

There was a interesting problem happend on Microsoft cluster when I came back from hospital. Our DBA team complaint Microsoft Cluster Service failed intermittently on virtual machine. This situation constantly happend for a week.

At the beginning of the whole troubleshooting, team noticed quorum disks failed with following Windows event:

Cluster service failed to update the cluster configuration data on the witness resource. Please ensure that the witness resource is online and accessible.

So we focused on disk performance. vbod.log also show some performance degrading but the time was not match. Microsoft was involved after that, they said the cluster failure actually caused by network connectivity issue according to following Windows event:

Cluster node ‘xxx’ was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

It became interesting since virtual machines share physical network links, it cannot be only single virtual machine had problem if there was network connectivity issue. Then we noticed there was following abnormal Windows event when some failure happend:

After involved multiple vendors from OS, virtualization, network and storage team, everybody said it’s not their problem. You could see this kind of problem in large datacenter since more and more system installed, it’s hard to find out which piece of the system caused the issue. You have to familar with each field of datacenter.

Eventually we figured out the issue related to storage workload. But why vendors cannot figured out this problem? First of all, Windows OS disk is running on shared storage, Windows no responding when the VMFS5 datastore latency of OS disk is high. From Windows perspective, it doesn’t know what happend on backend storage, it just know OS is very slow for few seconds, kind of pause the system. So it leads to network packages drop, and no Windows event for that since OS resumed very quickly, Windows takes it as normal behavior. Cluster actually failed at this moment. Secondary, the particular LUNs hosted the virtual machine was not busy, but the LUNs shared same storage pool with other LUNs. Any high workload LUN will impact rest of LUNs in same storage pool.

After understood these points, we figured out lot of virtual machines got high latency or high IO around 7PM every day, and most of the cluster failure happend this time. Since it’s impacted to large number of virtual machines, it must be caused by some common components on virtual machines. We eventially figured out it’s McAfee DAT updating after captured network packages. All virtual machines did same downloading in same time lead to high workload on shared storage and lead to cluster virtual machiens no responding for few seconds. The issue got fixed after change McAfee DAT updating schedule to random interval.

There are always common things going on on datacenter, it maybe small resource consumer but it can be a signaficant big monster in virtualized datacenter. Such as backup, monitoring, anti-virus or system management agents. It can impacts to shared storage or network links.

CSV (Cluster Shared Volume) is fundamental of Microsoft Hyper-V. You must have it to leverage Live Migration and High Availability features. But it’s very confuse when you want to reclaim CSV since CSV is using different name with physical disks. For example, CSV name usually is “Cluster Disk x”, path usually is “C:ClusterStorageVolumeX”. But real disk name is “Disk x” in Disk Manager. You have to very carefully when delete the disk.