Wednesday, 26 November 2014

Within the last week I have replaced 4 old virtual hosts with new Dell R720s. Initially everything seemed to be working fine but this morning we started to see LUNs disappearing from the Storage tab and dropped connections to Management ports. We also experienced large amounts of latency when attempting to manage the environment.After working with our storage supplier we determined that the problem was not on the Nexenta devices but elsewhere in the infrastructure.After a lot of testing, searching and head-scratching I discovered that there is a known issue with the Broadcom network cards and ESXi 5.x. The R720s have Broadcom NetXtreme BCM5720 network cards and onboard connections.The simple fix for this is to update the Dell pre-installed network drivers - in my case I went from 3.123c.v55.5 to the latest available version.First I checked my current version by SSHing onto the vHost as root and running the commands;lspci | grep BCM5720 (this displays all connected vmnics)0000:01:00.0 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic0]0000:01:00.1 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic1]0000:02:00.0 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic2]0000:02:00.1 Network controller: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet [vmnic3]

then;ethtool -i vmnic0After running ethtool you should get a response that looks like this;driver: tg3version: 3.123c.v55.5firmware-version: FFV7.10.17 bc 5720-v1.34bus-info: 0000:01:00.0I then downloaded the new driver package from the VMware download centre at https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI5X-BROADCOM-TG3-3137DV501&productId=285There is a ReadMe document in the .zip download but for ease of reference I have reproduced it below;COMMAND LINE INSTALLATIONNew Installation----------------For new installs, you should perform the following steps:1. Copy the VIB or offline bundle to the ESX server. Technically, you can place the file anywhere that is accessible to the ESX console shell, but for these instructions, we'll assume the location is in '/tmp'. Here's an example of using the Linux 'scp' utility to copy the file from a local system to an ESX server located at 10.10.10.10: scp VMware_bootbank_net-driver.1.1.0-1vmw.0.0.372183.vib root@10.10.10.10:/tmp2. Issue the following command (full path to the file must be specified): esxcli software vib install -v {VIBFILE} or esxcli software vib install -d {OFFLINE_BUNDLE} In the example above, this would be: esxcli software vib install -v /tmp/VMware_bootbank_net-driver.1.1.0-1vmw.0.0.372183.vibNote: Depending on the certificate used to sign the VIB, you may need to change the host acceptance level. To do this, use the following command:esxcli software acceptance set --level=<level> Also, depending on the type of VIB being installed, you may have to put ESX into maintenance mode. This can be done through the VI Client, or by adding the '--maintenance-mode' option to the above esxcli command.Upgrade Installation--------------------The upgrade process is similar to a new install, except the command that shouldbe issued is the following:esxcli software vib update -v {VIBFILE}oresxcli software vib update -d {OFFLINE_BUNDLE}VUM INSTALLATIONThe VMware Update Manager (VUM) is a plugin for the Virtual Center Server(vCenter Server). You can use the VUM UI to install a VIB by importingthe associated offline bundle package (a ZIP file that contains the VIB and metadata). You can then create an add-on baseline and remediate thehost(s) with this baseline. Please see the vCenter Server documentation formore details on VUM.I used WinSCP to copy the .vib file to the /tmp/ location on the host. After upgrading you will need to reboot the host.After following these steps, the network issues were resolved.

Monday, 7 July 2014

We encountered an issue this morning where our Kayako Case (formally known as Kayako Resolve) was repeatedly logging a ticket from it's email queue as the email was not being deleted. When manually running the email parser cron job I was seeing the following error;

After speaking with Kayako support it was determined that the problem lay with our Exchange 2010 server.On checking the error logs on Exchange I found an Application Error event 12014, under Task Category TransportService;

Microsoft Exchange could not find a certificate that contains the domain name name.domain.local in the personal store on the local computer. Therefore, it is unable to support the STARTTLS SMTP verb for the connector CONNECTOR-NAME with a FQDN parameter of name.domain.local. If the connector's FQDN is not specified, the computer's FQDN is used. Verify the connector configuration and the installed certificates to make sure that there is a certificate with a domain name for that FQDN. If this certificate exists, run Enable-ExchangeCertificate -Services SMTP to make sure that the Microsoft Exchange Transport service has access to the certificate key.

As we recently had to change our email SSL certificate, I checked the Receive Connectors named in the errors and found that the FQDN provided in response to EHLO or HELO did not match the name on the new SSL certificate.After updating this connector, email began to flow in and out of our helpdesk again.

Friday, 9 May 2014

I encountered this issue with Dameware Mini Remote Control as I was fed up of always having to right click, run as different user whenever I wanted to install the service on a user's PC.To get round this I did the following;1. Right click the program and go to Properties2. In the "Target" box, before the path to the executable, paste the following;

C:\Windows\System32\runas.exe /savecred /user:domain\username

3. OK your way out and run the program. You should be prompted for your admin password the first time, but after that the software should just load.

Saturday, 15 February 2014

Having recently gone through the decision making process for replacing our existing desktop infrastructure, I put together a list of considerations/questions that helped me to make a recommendation for our new desktops.I've grouped the questions under rough headings - the list is not exhaustive, simply the questions I asked;

Applications

What
applications do the users need to run?

Do those applications lend
themselves to being run in a datacentre?Can the application run in
a terminal server environment?

Application
Virtualisation

Is it
technically possible to virtualise/stream the apps?Does everyone need the app
or is it only some users?Does the application need
to access locally stored data?Does the application need
to integrate with other applications? If so, where are they?Does data need to stay off
the client device?

Costs

What
licencing is required? How much will it cost?What end-user equipment is
required? Are there different options? How much will it cost?What back-end equipment is
required? Are there different options? How much will it cost?

Support

What
difficulties are there in providing remote support to users?What potential end-user
hardware issues are there? (Device fails/user reset to factory)

Long Term

How often will we need to refresh the hardware?How will we go about software updates and
patching?Network

In a multi-site environment, are the network links between
sites able to support everything we want the infrastructure to do?