I share my thoughts and experiences as a Systems and Network Engineer

Menu

Quickie post – I’ve ran into a handful of servers since SBS2008 that have their Sites certificate expired and this results in a handful of event log messages and other annoyances. It a customer actually uses the Sharepoint service built-in, obviously the site won’t work properly using HTTPS because … well, the certificate is expired.

Reissuing this is pretty easy using the following steps:

Open the IIS Manager on the SBS Server

Click on the server name in the left-hand side tree (see image @ #4)

Double-click SSL Certificates, which opens that section

Click New Domain Certificate as shown:

Enter information on the certificate noting that the Common Name should be Sites:

On the next screen, select the Domain’s certificate authority and enter a friendly name:Note: To my knowledge, all SBS services come with AD Certificate Services installed, which is the basis for this whole article

The certificate is now available to IIS. Review your IIS Site bindings and move any sites using the old Sites certificate to the one you just creased.

Don’t forget to delete the old Sites certificate off the machine or you’ll get warnings in your Application log about it being expired.

For whatever reason, some Sonicwalls will not communicate with some Domain Controllers by simply selecting “Use TLS”. I’ve had some work just fine with this (and using the StartTLS verb), while everyone once in a while I see errors like “LDAP Communication Error” or “Error initializing SSL/TLS, data 0, v1db1:”.

I’m sure if I had more time I’d know the root cause, but here’s the fix.

On the Server, install the Active Directory Certificate Authority role and just the IIS Management Console (no need for the IIS service itself).

On a typical DC the defaults for the AD CA setup are fine – store the authority in AD.

Again, just install the IIS Console – no need to install IIS and add attack vectors!

Open the Certificates console by opening MMC.EXE, going to File -> Add/Remove Snapin and selecting Certificates. You’ll want the Local Machine’s Computer store.

Save to a file on the desktop, etc – use the .P7B format to make the Sonicwall happy

I selected the “Include all Certificates in Chain” option.

Open the IIS Console and click the server name. There will be only one or two options, one will be for Server Certificates. Double-click that to open.

Click the Create Domain Certificate link on the right. Enter information for a Sonicwall-specific certificate. Select Next and select the domain’s certificate authority.

Once created, go back into the Certificates console and right-click the certificate, exporting the certificate with the private key. You’ll need to set a password to protect this key (it’s temporary since you’ll want to delete this file at the end of this process)

You now have the CA Certificate and a signed certificate for the Sonicwall using that CA.

Log into the Sonicwall. Go to System -> Certificates.

Click the Import button, select the option to import a CA. Select the file created above and import the CA.

Click the Import button again. Select the option to import a certificate. Enter the password used above to protect it, as well as a “Friendly Name” and the certificate file itself.

Confirm both the CA and the certificate show up in the Sonicwall’s page.Important! Make sure the certificate shows a Validated status of Yes – otherwise it won’t be selectable in the next steps.

I have a customer who utilizes a Zix appliance for their email. It looks like an interesting appliance. Unfortunately, it did not come with a known password for the web console.

I searched around for a reset procedure before calling their support. The reset process is pretty easy and seems pretty obvious if you’re familiar with initial setup procedures, so I’m going to share it here hoping to save someone time. This assumes you have the root password for console access to the device itself, if you do not have this I’d suggest calling their support – they’re pretty friendly people. It looks like it would be a simple Linux root password reset, but I’m not that familiar with the device.

Having said that, it’s simple. Log into the unit and run the zixconfig command:

Select the option to add/remove users:

And then add a user:

Why this isn’t on their website – I can’t say. Would probably save everyone some time.

I’m working an issue with a customer at the moment who contracted a computer virus which spread through their whole network, killing all machines on it. This includes servers and workstations.

Here’s what I can tell you:

The virus itself appears to be present in the file c:\windows\samsam.exe

A process monitor shows when it runs, it creates a file in c:\windows\prefetch (presumably to run itself on startup) as SAMSAM.EXE-<RANDOM>-PF

It enumerates your directories and then begins encrypting files using the Windows Cryptography API.

It creates the file c:\windows\system32\selfdel.exe

It creates c:\windows\del.exe, which is a copy of the SysInternals SDELETE tool. It then runs the command “c:\windows\del.exe -c C: /accepteula” to wipe the free space on the disk, preventing recovery.

Once this is done, it then wipes the samsam.exe application off disk.

Unlike many variants of Cryptolocker, this one spreads through some yet-to-be-determined manner to other machines on the network. I have it disconnected and thus cannot see how it connects to other machines (or, if it perhaps just overwrote an executable on a network drive).

Unlike any variant of Cryptolocker I’ve seen – it spreads to servers and runs on them as background processes, even stopping the Exchange databases and attempting to encrypt them (however, on this customer’s >1TB datastore, it just renamed it – perhaps it renames, encrypts to a temporary file and then copies over?).

It will break the Symantec Endpoint Protection install on the machine to an unusable state.

It is (as of this immediate moment) not detectable using Symantec Endpoint Protection or Webroot.

I am opening a ticket with Symantec and working it to provide them with the executable and process traces to get this puppy identified but for now – no Symantec tool we’ve ran picks it up.

I love SonicPoints! They are one of my favorite local wireless solutions and I’ve found them to be pretty darned reliable over time. But like anything, issues can creep in causing problems with your setup. Having said that, here’s my picklist of typical fixes for Sonicpoint issues.

PoE Issues: The older Sonicpoint devices are 802.3af/PoE/15.4w devices, whereas the newer Sonicpoint devices are 802.3at/PoE+/25.5w compliant. This is often overlooked and they’ll power up on a lower-powered port anyway, but often will reset frequently as the power draw approaches the switch maximum. Make sure you’re using the proper injector/switch for the device. Also – remember that longer runs means more voltage drop.

Spanning Tree: When a switch port becomes active, the spanning tree protocol will discard traffic until it determines the STP state of the connected device. For some switches, this can mean a fifteen second delay (or more) between when the device sees the link active and when traffic actually passes over it. Sometimes, the SonicPoint doesn’t get a reply from the Sonicwall fast enough and fails. Put the port in portfast mode (or turn off spanning tree for that port).

EEE/Energy Efficient Ethernet: I know this is an odd mention, but many switches have EEE turned on by default when shipped and it’s not something many people think about, but the switch will scale the transmit voltages to what it thinks is the minimum required to work. Sometimes, it guesses wrong and you end up losing carrier.

L2/L3 Issues: The L2/L3 network available to the Sonicpoint at boot-time should be dedicated to SonicPoint management (i.e. there should be nothing else on the network). On my devices, I dedicate a physical port to the SonicPoint traffic (to avoid bandwidth competition with the LAN), carry the untagged traffic on that interface to the SonicPoints and VLAN off each SSID. That way, devices aren’t messing around with the SonicPoint management traffic on either an L2 or L3 level.

Firewall Rules: There are some hidden assumptions in the Sonicwall firmware, like preventing wireless devices from talking to one another. When in doubt, check the Packet Monitor to ensure traffic is not being blocked by a firewall rule.

DHCP: Twice encountered, it’s best to let the Sonicwall do any DHCP relaying over your network switch(es). I’ve seen issues with traffic not generated by the Sonicwall not making it to the client (which is configurable in the Sonicwall but a caveat nevertheless).

Don’t be afraid to factory default. I’ve had SonicPoints go awry and after numerous power-cycles, a factory default left it resolved.

Virtualization is a complicated subject that is not well understood even by IT professionals;

Virtualization has been made available to the common man without any barriers to entry.

Following this, I thought I might make a quick “how-to” guide for people who might have dipped their toes already, but want to know a bit more in terms of “best practices” for smaller HyperV (or VMWare!) setups. It’s a collection of my own personal best practices that I hope may help someone else. I hope to expand it as I learn myself.

Networking

I’ve not recommended or installed a server with less than four Ethernet ports in years. Having said this, one common setup mistake I see is a single Ethernet cable hanging out of a server, the others screaming to be used. Here are a few recommendations:

If your server is using iSCSI to connect to a SAN, make sure you have at least two interfaces worth of traffic to/from your SAN. Read about VMWare MPIO here in this PDF or Hyper-V MPIO here.This gets you mutiple paths to your storage increasing total overall bandwidth to your SAN and giving you the redundancy of two networking paths.

If you are using iSCSI to connect to a SAN, enable jumbo framing on your switch. After confirming your switch supports it, enable the feature (usually a global option and usually requiring a switch reboot). For the NICs involved (SAN and Server) change the MTU to 9000 (or whatever the least common denominator is for your equipment). On some Windows NICs you may have to enable other features in the NIC driver settings.For most workloads, this gets you faster speeds and lower latencies on your Disk I/O.

Make use of NIC teaming for your VM connections – and disregard the “One NIC per VM” ideology unless there is a very, very specific reason to do so. Teaming four NICs together for VM I/O means each VM is sharing 4Gb/s. It means, for example, your File/Print VM could get up to 4Gb/s to your LAN – but it also means that an unplugged Ethernet cable isn’t going to bring down any VMs. Typically, this means simply adding physical interfaces to your virtual switches. Be aware that this is like having (4) 1Gb/s highways, not one 4Gb/s highway – no individual car is going faster than 1Gb/s, but cars are automatically distributed to the faster highway depending on where they are coming from/going to.It’s the cheapest, easiest way possible to increase Network I/O and redundancy. Seriously.Recommended Reading: VMWare Teaming, HyperV Teaming, Some Basic LACP vs. Static Info

Before you use SR-IOV or other networking virtualization technologies, understand what they do and know they aren’t a panacea. Many people I’ve worked with will turn these babies on and then – something doesn’t work right or they aren’t able to connect to the LAN. There are specific needs when implementing SR-IOV and unless you can state them all and understand them, don’t turn it on. It’s also worth mentioning that these technologies really only kick in around the 4-5Gb/s range, so unless you’re pushing that traffic continuously out your VMs, it’s not likely to yield a large benefit.Don’t use SR-IOV or other advances unless you know exactly why you’re turning it on.

Always, always install the latest drivers to your physical system. For example, Broadcom NICs have an issue with VM Queuing (‘VMQ’) that could cause high latencies, packet loss or speed issues between your virtual switches and your external network (see here). These new technologies mean new bugs, so keep things up to date and save some time.

Storage

Most setups in my scope aren’t using SANs, but local storage on disks in the server chassis. This is fine for many setups, especially small businesses who are consolidating a few server boxes into one setup for cost savings. I fully expect some of my recommendations to flame a few people off, but again – I base this on my experiences fixing other people’s mistakes.

Never, never, never use dynamic disks unless you fully understand a few things:

They will fragment your local storage significantly. As the VHD files grows, they consume blocks all over the physical disks, Internally, 100MB of contiguous file space is written all over the file as well, meaning your performance will only reduce over time.

They are slower – because of the information above. As updates are installed, files written and changes made, the local drive heads have to flicker all over the disk to read.

They are tough to plan for. Seriously, seriously tough. I’ve watched people ignore my advice, build 10 VMs and tell me how wrong I was, only to ask for my help two years later when the physical disks slowly filled to my favorite 0 Bytes Free status. Then you’re shrinking dynamic disks onto USB drives, then copying them back while the customer is down, and telling them they need to order two more disks for something you should have planned for 2 years ago.

The best intentions of your VM OS (trying to defragment their disks, or allocate files in long contiguous chains) will affect your physical dynamic disk in the exact opposite manner – stretching out the list of non-contiguous blocks even further.

Snapshots make the matter worse. Snapshots – which are essentially dynamic disks themselves – require all reads to go through two dynamically mapped disks.

Backup solutions running on the physical host(s) require the disks to simmer down for a few seconds while they can do a snapshot take much longer with dynamic disks.

Converting them to Fixed disks, even if you plan to do it later, require the VM to be off. So, a lot of people just forget to do it – don’t be those people.

Separate your system disks onto a smaller RAID1 array, where you keep your Windows installation and any other software that might compete for Disk I/O (you’re not really going to install anything else on a HyperV machine, right?). Put your VMs on their own disks with faster spindles and dedicated I/O.

Don’t skimp on your RAID controller. Seriously, I’ve seen people order low-end RAID cards with no cache memory and drop four disks on it in RAID5 mode – and then act surprised when I/O stinks. Cache memory means the card can wait to write to disk when it’s busy and a cache battery means it can do it safely. Don’t cheap out on these. In fact, just don’t use a RAID card that has less than 512Mb of RAM and a backup battery.

If the goal of your projects is to get your servers all under one chassis, don’t be afraid to install that separate RAID array for your Exchange databases or SQL data. If your storage is busy, it’s better to have your Exchange database on it’s own RAID1 array than to constantly lock up your shared storage.

Avoid using pass-through disks for production data unless you have a good reason. I’ve seen this one but talked about many more times, usually with the above rule – someone virtualized a database application and thought the SQL volume should go on passthrough disk (“it’s faster, man!”). It’s about 2% faster with the oldest, least optimized technology. Tell me it’s faster when you have to migrate it somewhere else.

Memory/CPU

This section is much harder to understand – it touches more deeply in the hardware than you might be used to. It’s worth noting that for most people, they assign what they think the correct amount of RAM and CPU cores is necessary for a workload and then dust their hands clean. That’s not a smart idea. Check out this rough diagram of an HP ProLiant DL380 G8 I just installed for a customer:

Let’s imagine you undersized the host and later installed a VM that required more resources. For example:

Your VM requires more than 16GB of RAM;

Your VM requires more than 4 CPU cores;

Your VM requires 8GB of RAM, but you have two other VMs running using 10GB each.

In each of these situations, you’d be doing what’s called NUMA Spanning. This means that the memory or core count allocated to your VM cannot be assigned to either the free RAM connected to one CPU in whole or is assigned more cores than a CPU can offer, meaning the VM is running on two different CPUs.

This means that the CPU running one or more threads comprising your VM has to stop and ask the other CPU to read/write the remote memory connected to it – a huge performance penalty.

Often, when one VM host is sold, the bare minimum number of CPUs, cores and/or RAM is installed and your VMs cannot fit neatly into one NUMA node or the other. Sometimes it won’t affect the VM enough to matter, while other times it cripples performance. You can turn the feature off but if you do – beware – for your VM won’t start unless it can fit neatly in one NUMA node!

Check out the Wikipedia entry for the NUMA topology here, but know it’s not incredibly specific in regards to the Windows implementation. How it affects HyperV can be found here, and VMWare’s more technical and official dive is found here.

I’ve spent a lot of time recently doing P2V conversions to HyperV hosts. For each server, I performed a P2V and after installing the HyperV integration tools, I received a Windows Boot Manager error on file Vid.sys with Status 0xc0000098 and an error message Windows failed to load because a required file is missing, or corrupt as shown on next reboot:

The immediate resolution is (please see below, however):

Shut off the VM and mount the System VHD/VHDX file in the HyperV host machine;

Browse to <Drive>:\windows\system32\drivers and rename vid.sys to vid_old.sys;

Unmount the drive from the host and start the VM;

Remove the device from Device Manager by going to View > Show Hidden Devices, then opening Non Plug and Play Devices and removing the Vid driver.

Now, I don’t advocate randomly deleting device drivers, but I was completely unable to get my servers to start any other way. I’ve had this happen with three different machines each having the same resolution. Here’s what I’ve found:

All servers involved have been converting from Dell PowerEdge servers – this may be circumstantial;

I’ve not had the issue with newly-built Server 2008R2 SP1 VMs or machines converted from HP ProLiant hardware, each with the same versions seen below;

For all converted machines, I used Disk2VHD or ShadowProtect to create the VHD file. On one server, I attempted both suspecting an issue in conversion, both had same issue;

On each server, I mounted the VHD and performed a CHKDSK to be sure there was no filesystem corruption – none was found;

On all servers (affected or unaffected), the VID.SYS file had a version of 6.1.7600.17514 (from Server 2008R2 SP1) and an MD5 hash of 1720d283bdb1eaa7f21976586ff52b95.

However, a Server 2008R2 non-SP1 server converted from the exact same Dell server as an affected server did not have the issue with VID.SYS version 6.0.6002.18005.

On one customer server, a P2V conversion had the same issue, but two fresh Server 2008R2 SP1 installs on the same host with the same integration services installation did not have the issue with the same VID.SYS driver file;

The Integration Services installation cabinet file does not contain a VID.SYS image, and so I have to assume it’s not actually touched by the installation software.

None of the servers converted had HyperV installed on them prior to (or after) the P2V.

All of this information leads me to believe that the issue has nothing to do with the VID.SYS file or the integration tools, but some other interaction I have not been able to sniff out. However, the purpose of the VID.SYS driver is to facilitate communication between the Parent OS installation and the Hypervisor software, which isn’t necessary for any of my VMs (since they aren’t running HyperV in that manner). Thus, I feel it safe to leave the driver off of the system.

A quicky, but something I don’t think many people know about. If you’ve ever had a MySQL server that seems slammed, but a SHOW PROCESSLIST shows nothing of note, it’s likely because your server is keeping up with processing the queries and not hanging up on them – which is good because that means your server can keep up with the load, but bad because the query isn’t “hanging out” in a non-completed status and showing in your list.

If you want to “dip a ladle” into the queries for a few moments and see what’s being processed, log into the server with root (or equivalent privileges) and issue the following commands. Please note that these entered use around MySQL 5.1 (check here for more information):

MySQL will happily log the queries entering the server for you as long as you want it to, to give you a chance to see what’s got the thing so busy. If you’re not the only admin on the server, you might want to issue a SHOW VARIABLES command to see if someone else is already doing some logging to avoid stepping on their toes.

Working as a consulting engineer means frequently walking into situations you know nothing about … and equipment you have no access to. Today’s example was walking into a small business telco closet and staring down a phone system and a 48-port switch which the phone vendor needed reconfigured.

Of course, they didn’t know the password…

Here’s how to reset the password on the V1910 and not totally lose the configuration (read through for the caveat) – unfortunately this was an emergency situation and I have no V1910 of my own, so I have no screenshots to accompany.

Attach a console cable and confirm communication (typically these devices are set to 38400 baud, 8/N/1).

Power-cycle the device and watch for a boot prompt offering options if you hit CTRL-B. Oblige it.

Select option #7 to boot without a configuration file. This doesn’t delete the configuration, it just tells the switch to ignore it.

Reboot the switch as it suggests.

Once it gets to the command prompt, start the ‘hidden’ command prompt by entering _cmdline-mode on and hitting enter. It’ll ask you if you want to do this, you do. Enter a password of 512900 to gain access.

Once you are in, be aware that there are two modes – user and system – something like the standard and enabled prompt on a Cisco. You can enter the system configuration by typing sys and hitting enter, but you begin in user mode. Here you can type more flash:/startup.cfg to see the existing configuration. There’s also an XML file on some switches – review this file too! For the curious, you can dir the contents of flash just like a Cisco.

If you are lucky, the admin user’s password was just revealed to you in plaintext in the configuration file. You could reboot now and try entering it.

If you are unlucky, you could copy the configuration file off of the switch using the backup and restore commands in conjunction with a TFTP service and delete the local-user admin sections to leave it as default.

On some switches, I’ve had to use the startup command to tell the switch which startup file to use upon next reboot. I might do this here for safety.

Pretty simple – I have yet to see a how-to guide that doesn’t tell you that an initialize command is required, so I’m hoping this helps someone!

Taking a break from my techie writing (and the endless yet unanswered temptations to wax political) I’ve decided to talk about music – one of my favorite things. On any other day, I could be talking about my classic rock collection on vinyl, Pink Floyd lyrics or Led Zeppelin guitar riffs. Today I’m talking about another genre I love, Ambient Trance.

Here are some of my favorite songs – none contain lyrics, because they are good enough to convey emotion without needing them. While some of these songs are pretty different, I can’t help but feel they are more than the sum of their parts. I hope to make this a constantly updated list as I remember and find more!

Boards of Canada – Pete Standing Alone

Not just because my name is in the title, but this song has a simple, subtle melody to it. It has parts that uplift and parts that make you introspective.

Boards of Canada – Everything You Do Is A Balloon

Another Boards of Canada song – I love these guys. It takes a while to get started, but lay back on your couch and close your eyes while listening to this song – I bet you’ll have some amazing memories come back to this wonderful soundtrack.

Richard Devine – Randale

I have to admit, I’ve not really fallen in love with much of his other stuff. But this song is … an amazing journey. It reminds me of my first heartbreak on one hand, and a long weekend as a child playing at a beach-front campground on the other.

Chicane – Offshore

I could include almost everything ever done by Nick Bracegirdle! This song has been listened to for almost twenty years of life and it still holds the same amazing feeling that it did in 1997.

Moby – Everloving

Moby has a special place in my heart. He’s such a good musician, truly understanding the human connection with the art. This song tears at my heart like flying away from a loved one for the last time, and warms it at the same time like a sunset on the ocean.

Carbon Based Lifeforms – MOS 6581

I dare you to listen to this song and not question everything you know about life.

Symbion Project – Our Breath Shall Intermix

First, I’ll openly admit this isn’t like the others. It takes forever to really start, it’s very slow paced and is almost three different movements in one song. But I’ll be damned if it doesn’t feel like I’m being ripped out of my chair and shown something amazing.

Royksopp – Royksopp Forever

Royksopp does some really, really cool stuff. This song blows their normal work out of the water. An epic summary of just about every gratifying or painful life event I’ve experienced.

H.U.V.A. Network – Cobalt

I discovered this song thanks to Pandora on a long nighttime drive about 6 months ago. Significantly more clubby than my other selections, this song definitely imparts a very cool, calm and mellow energy on you when listening.