Tag: vmware

Here’s a quick guide to run an NTP (Network Time Protocol) server using Chrony with a GPS (optional) receiver on a VMware ESXi Guest running Ubuntu 18.04. I should note this is experimental and something I setup in my homelab temporarily. For production environments I would run NTP on physical hardware and not VMware.

Create and Configure VM

Be sure to disable Guest Tools Time synchronization by editing the VM settings and uncheck Synchronize guest time with host.

Set the CPU shares to High… we want the NTP server to have priority if there is processor contention.

Install Chrony

Shell

1

2

sudo apt install chrony

sudo vim/etc/chrony/chrony.conf

/etc/chrony/chrony.conf

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

# Welcome to the chrony configuration file. See chrony.conf(5) for more

# information about usuable directives.

# This will use (up to):

# 3 sources from ntp.ubuntu.com

# 3 sources from us.pool.ntp.org

# 3 sources from time.nist.gov

pool ntp.ubuntu.comiburst maxsources3

pool us.pool.ntp.orgiburst maxsources3

pool time.nist.goviburst maxsources3

# This directive specify the location of the file containing ID/key pairs for

# NTP authentication.

keyfile/etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate

# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.

rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than

# one second

makestep1-1

allow

I diversified between Ubuntu’s, NTP.org’s and NIST’s time server pools.

That’s it, after restarting the chrony service (service restart chrony) you should be able to get time reports by running:

Shell

1

2

3

chronyc sources-v

chronyc sourcestats-v

chronyc tracking-v

Why You Shouldn’t Run an NTP Server in a VM Guest

VM’s can’t keep accurate time

I’ve generally found that VMs keep great time inside of VMware. One thing that can help with this is setting the CPU shares to high so your time server always has a priority. I ran Chrony in a VM for several weeks, compared it with Chrony on a Raspberry Pi. Both were acceptable, and both had a smaller standard deviation than public NTP servers over the internet, but the VM had a much smaller standard deviation than the Pi. That tells me VMs running on better hardware may be better than lesser bare physical hardware at time tracking under certain conditions, and a local NTP server in a VM can be more precise than grabbing time off the internet.

VMs can become out of sync during snapshots, suspend, failover, etc.

I ran a suspend test and this is true. I paused a VM, waited 10 seconds, then resumed it. It reported the wrong time to NTP clients for several minutes before it corrected itself from external NTP servers. Here’s a screenshot of my NTP server being 11 seconds off after a pause!

This is a valid reason to run an NTP server on physical hardware. However, I think it is possible to run an NTP server under VMware with the following precautions:

Your NTP servers under VMware should never be paused. That means they should be excluded from failover (instead of failover it’s better to configure multiple NTP servers for your clients to connect to since it’s better for an NTP server to be down than report a wrong time).

Have multiple NTP servers. At least three. You’ll notice in the screenshot above Chrony (running on a separate physical machine) flagged the server as not being accurate. This way if one of your VMs gets paused chrony will switch to another time-source automatically.

Set makestep 1 -1 in the chrony.conf file (this tell chrony that any difference greater than one second will get stepped which allows for faster correction after a resume).

GPS Receiver

This is not really related to VMware. But I had a GPS receiver so thought I’d see how it works with Chrony….

I have a GlobalSat BU-353S4 USB GPS Receiver. This isn’t the best GPS receiver for accuracy. For me it’s accurate to within a few hundred milliseconds which is good enough for my experimental purposes but worse than just grabbing time off the internet. For serious time-keepers you’ll be wanting to use something faster than USB and more accurate than what a cheap GPS receiver can provide.

Configure gpsd

Shell

1

2

sudo apt install gpsd gpsd-clients

sudo vim/etc/default/gpsd

/etc/default/gpsd

Shell

1

2

3

4

5

6

7

8

9

10

11

12

# Start the gpsd daemon automatically at boot time

START_DAEMON="true"

# Use USB hotplugging to add new USB devices automatically to the daemon

USBAUTO="true"

# Devices gpsd should collect to at boot time.

# They need to be read/writeable, either by user gpsd or the group dialout.

DEVICES="/dev/ttyUSB0"

# Other options you want to pass to gpsd

GPSD_OPTIONS="-n -F /var/run/gpsd.sock"

Install Chrony

1

2

sudo apt install chrony

sudo vim/etc/chrony/chrony.conf

/etc/chrony/chrony.conf

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

# Welcome to the chrony configuration file. See chrony.conf(5) for more

# information about usuable directives.

# This will use (up to):

3sources from ntp.ubuntu.com

3sources from us.pool.ntp.org

3sources from time.nist.gov

1GPS device

pool ntp.ubuntu.comiburst maxsources3

pool us.pool.ntp.orgiburst maxsources3

pool time.nist.goviburst maxsources3

refclock SHM0refid GPS precision1e-1offset0.250delay1e-9

# This directive specify the location of the file containing ID/key pairs for

# NTP authentication.

keyfile/etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate

# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.

rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than

# one second, at any time

makestep1-1

allow

So, how did I get the values on the refclock line…

The way I came up with my offset of 0.250 is by initially setting the offset to 0.0, restarting chrony, and running chronyc sources -v several times taking note of the offset. I’d get numbers like +249ms, +253ms, +250ms, etc.

Since my GPS is off by about 250ms I set the offset to 0.250. Now it’s usually not off by more than 100ms.

The 100ms+- variance is not a problem when being combined with other sources, but if it was the only time source I’d be better off tolerating drift than the high variance of GPS for a short period without access to the NTP pools, if I had no internet for several months or an air-gapped network then time via GPS would probably be better than nothing–but a better GPS receiver should be used in those scenarios.

For most networks running chrony in a VM and using a GPS is unnecessary. It’s better to keep it simple. I just use the NTP service on my pfSense router and set all the clients to that.

Why You Should Have a Homelab

In 1998 my friend gave me a RedHat Linux CD. I spent hours each day experimenting with Linux–I loved it. 2 years later I’m in a room with 30 other students at a University applying for the same computer lab assistant job–I’m thinking my chances are grim. Part way through the mass interview a man walks to the front of the room and asks if anyone has ever used Linux. I raise my hand–I’m the only one. He takes me out of the interview for the lab assistant job, introduces me to the department director. They took me out to lunch. By the end of the day I had my first job as a Systems Administrator.

Learn things on your own and it will broaden your opportunities.

One of the best ways to learn about systems, applications, and technology is starting a homelab. A Homelab can give you an enjoyable, low stress, practical way to learn technology. A homelab will also help you find out the technical areas in which you are interested. It’s also practical in that you can use it to service your own home.

Here’s 7 Ideas for Your Homelab

1. Router / Firewall

The most essential piece of equipment will be your router. I started out with consumer routers that I’d flash to DD-WRT / Tomato but now I use a virtual pfSense router. Routers are great to learn about DHCP, DNS, VPN, Firewalls, etc. I discourage using the router provided by your ISP, they’re usually not very capable and often not secure. In most cases you can buy a DSL or Cable modem instead of the ISP provided modem combined with the router. One inexpensive physical router I’d recommend is the Ubiquiti EdgeRouter X. Ubiquiti provides free software updates (their model is you buy the hardware and the software is free), and you’ll get a handful of advanced features–it’s a very capable router and much better than a typical consumer router–to step up from Ubiquiti you’d be going to pfSense, Juniper or Cisco.

2. Storage

The main reason I started my homelab was storage. I was taking a lot of family pictures and videos and wanted to save them. I know there are cloud services, but at the time they were expensive, and then you’re sort of trusting that provider to not delete all your photos or get bought out by a larger company and shutdown.

Then I started using VMware. I needed faster storage with more IOPS. One of the best Homelab storage solutions is ZFS. ZFS takes the best of filesystems, and the best of RAID, and combines them into a software defined storage solution that I’ve not seen any hardware technology able to match. Two popular free ZFS appliances I like are Napp-It (based on OmniOS) and FreeNAS. OmniOS is a fork of OpenSolaris and is very robust and has tight integration with ZFS.

I’m currently using FreeNAS which is the free open source version of iX System’s TrueNAS which is used by organizations of all sizes–from small businesses with a few TB of storage to large government agencies with PBs of storage. FreeNAS has done a great job at technology convergence. It is both a NAS and a SAN allowing you to try both approaches to storage (I prefer NAS because it takes better advantage of ZFS, but many prefer using SAN and there are benefits and drawbacks to both), it also has many built-in storage protocols: FTP, iSCSI, NFS, Rsync server, S3 emulator, SMB (Windows file server), TFTP, WebDav, it can join AD, it can even be an AD DC (if you like living on the edge) it has a built-in hypervisor (bhyve) to run VMs for whatever you want. This is now marketed as hyper-converged storage. All of it is completely free. You can build your own FreeNAS server like I did, or get started with a FreeNAS Mini from iX Systems.

A few years after I learned ZFS for home, my employer was looking for a new storage solution so having this knowledge and experience was helpful. I was able to determine one vendor with a traditional RAID solution didn’t handle the RAID-5 write-hole problem properly.

3. Virtualization

Virtualization allows you to run multiple virtual servers on the same piece of hardware. VMware is king in the small to mid-size business hypervisor market, and VMware offers their hypervisor for free. The free version is just like the paid versions except you won’t be able to use some features (most involving high availability and fail-over with multiple servers). But you can learn most of the concepts and features of VMware. I’ve tried to use a number of hypervisors but I always come back to VMware. I consider VMware my basic infrastructure. From there you can learn about other things like networking, storage, and play with any OS or Linux distribution you want to.

Knowing VMware was hugely beneficial, I’ve implemented it for several businesses, and one of my previous employers. And knowing how it works means I can discuss the VMware stack intelligently with the ops team.

4. Networking

A Homelab without decent networking won’t get you far. Fortunately if you use VMware you can leverage it to use virtual network switches. For physical switches I really like the Unifi products. They are simple enough for non-network engineers like me. Everything can be configured using the GUI. Unifi exposes you to managed switches, central management (with the Unifi controller), VLANs, and PoE (Power over Ethernet), port trunking, port mirroring, redundant paths with spanning tree, etc.

I started with this little 8-port switch (4 are PoE ports). I also added a 24-port switch so I could learn how to do setup a LAG and configuring VLANs across multiple switches (which was really simple using the Unifi interface). I also like Unifi’s philosophy–they sell you the hardware but the software is free–which means you don’t pay for maintenance or support but continue to get free updates. In a homelab you may not need to go crazy on VLANs, but separating your main network from your IoT devices may be prudent.

Learning how to setup VLAN tagging, and link aggregation and understanding how networking works helps me communicate better with the network engineers when discussing design and deployment options–they may be working on Juniper or Cisco equipment but I know the concepts of what they’re doing.

5. Wireless APs

Having a robust wireless setup is also a necessity for a homelab. If you have a large house you get to setup multiple APs and make sure they can handoff connections. I use a Unifi AP Pro (I just use one because that’s all I need to cover my house, but if you can find an excuse to have 2 or more I’d recommend it since you can practice rolling updates without downtime, wireless handoff, etc.). These are managed by the same Unifi controller as the switches. I first gave them a try because I read Linus Trovalds uses Unifi APs, and they seem to be highly rated by tech professionals–and now I don’t think I’d go back to anything else.

6. Network Monitoring

It is hard to maintain a reliable network and application stack without monitoring for failures. There are hundreds of network monitoring solutions and it really depends on your needs. The most widely deployed solution is Nagios. I have had that on my Homelab, but lately I’ve been using Icinga because it’s simple and it integrates into Ansible.

7. Infrastructure Automation

Automating your infrastructure may not make as much sense in a small Homelab, but it does make sense to automate any task you do repetitively or a manual task that could be automated. For me, this was installing updates, deploying servers and renewing SSL certificates with Let’s Encrypt. To manage this I use Ansible which is one of the most well thought out infrastructure automation tools I’ve seen. Ansible can manage Linux and Windows servers. Learning infrastructure automation, especially if you do it using version control and CI/CD tools like Azure DevOps (you can get a free account for up to 5 users with unlimited private repositories) is a great thing to learn for your career if you’re interested in the DevOps world. The book, Ansible for DevOps by Jeff Geerling helped me get started. I suggest getting the eBook since he has been known to provide updates to the book (not sure if he will continue to provide updates, but just in case).

At work we completely automated the deployment of Linux servers using Ansible–infrastructure as code. It took a month of investment but it paid off big time with developers now being able to deploy VMware VMs at will with Ansible by making a Git Pull Request, our entire fleet of servers is updated automatically, and our server and configurations are all consistent. This replaced an old process of waiting several weeks for a VM to be provisioned and configured by hand.

Bonus homelab application server ideas…

Minecraft Server — popular Java game–it’s like playing with Legos and a great way to get your friends together for some casual games.

Mumble Server – one of the best voice protocols for in-game communication.

Emby Media Server — Anyone that has kids realizes those flimsy blu-ray drives aren’t going to last long. It’s great to store and host movies, home videos, pictures, and audio.

Asterix PBX Server – VoIP Phone server (use Twilio or Flowroute for SIP trunking). Polycom makes great VoIP phones. With Twilio SIP Trunking you can have a real landline phone number with E911 capability for a few dollars a month–and if you get multiple phones you can use it as an intercom system.

Web Server (maybe start a blog) — I hosted this blog from a server in my house for years–until my ISP couldn’t handle the bandwidth. Now days you can also use a service like CloudFlare to act as a CDN which really reduces your bandwidth usage. Hosting your own blog is a great learning experience and gives you a place to log your homelab experiments, and share solutions to problems.

Backup server — I use a CrashPlan Business subscription to backup my FreeNAS server to the cloud (one of the main reasons I use a NAS as this would be less efficient with a SAN). BackBlaze B2 is another great option to backup FreeNAS.

…

There are many more areas than I listed, but I think the above is a good baseline to get started. Pick one area at a time–my homelab was built over many years–often the case is I will improve an area after a piece of equipment fails or I need to replace it for some other reason–that’s a great time to do research. If you aren’t sure where to start, pick the area that you enjoy the most. For areas you have no interest the best thing to do is something else–you’re probably not going to be great at something you don’t enjoy. Certainly a homelab isn’t going to be a substitute for real-work experience. But it does provide an environment to learn, experiment and enhance your abilities–and the great thing is since it’s your own lab you can learn things that interest you.

I think that’s the largest benefit of a homelab. To me it’s a playground. It’s a place put the love of learning into practice. It’s a place of freedom. Nobody else is dictating what you do here. It’s a place to have fun while enhancing your skill.

Do you see a man skillful in his work?He will stand before kings;he will not stand before obscure men. – Proverbs 22:29 ESV

Hardware

4 x 2TB HGST RAID-Z, 100GB Intel DC S3700s for ZIL (over-provisioned at 8GB) on an M1015. In Environments 1 and 2 this was passed to FreeNAS via VT-d.

2 x Samsung FIT USBs for booting OS (either ESXi or FreeNAS)

1 x extra DC S3700 used as ESXi storage for the FreeNAS VM to be installed on in environments 1 and 2 (not used in environment 3).

Environments

E1. ESXi + FreeNAS 11 All-in-one.

Setup per my FreeNAS on VMware Guide. Ubuntu VM with Paravirtual is installed as an ESXi guest, on NFS storage backed by ZFS on FreeNAS which has raw access to disks running under the same ESXi hypervisor using virtual networking. FreeNAS given 2 cores and 10GB memory. Guest gets 1GB memory. Guest tested with 1C and 2C.

E2. Nested bhyve + ESXi + FreeNAS 11 All-in-one.

Nested virtualization test. Ubuntu VM with VirtIO is installed as a bhyve guest on FreeNAS which has raw access to disks running under the ESXi Hypervisor. FreeNAS given 4 cores and 12GB memory. Guest gets 1GB memory. Guest tested with 1C and 2C. What is neat about this environment is it could be used as a stepping stone if migrating from environment 1 to environment 3 or vice-versa (I actually tested migrating with success).

E3. bhyve + FreeNAS 11

Ubuntu VM with VirtIO is installed as a bhyve guest on FreeNAS on bare metal. Guest gets 1GB memory. Guest was backed with a ZVOL since that was the only option. Tested wih 1C and 2C.

All environments used FreeNAS 11, E1 and E2 used VMware ESXi 6.5

Testing Notes

A reboot of the guest and FreeNAS was performed between each test so as to clear ZFS’s ARC (in memory read cache). The sysbench test files were recreated at the start of each test. The script I used for testing is https://github.com/ahnooie/meta-vps-bench with networking tests removed.

No attempts on tuning were made in any environment. Just used the sensible defaults.

Disclaimer on comparing Apples to Oranges

This is not a business or enterprise level comparison. This test is meant to show how an Ubuntu guest performs in various configurations on the same hardware with constraints of a typical budget home server running a free “hyperconverged” solution–a hypervisor and FreeNAS storage on the same physical box. Not all environments are meant to perform identically…my goal is just to see if the environments perform “good enough” for home use. An obvious example of this is environments using NFS backed storage are going to perform slower than environments with local storage… but it should still at the very least max out a 1Gbps ethernet. This set of tests is designed to benchmark how I would setup each environment given the constraint of one physical box running both the hypervisor and FreeNAS + ZFS as the storage backend. The test is limited to a single guest VM. In the real world dozens, if not hundreds or even thousands of VMs are running simultaneously so advanced hypervisor features like memory deduplication are going to make a big difference. This test made no attempt to benchmark such. This is not an apples to apples test, so be careful what conclusions you derive from it.

CPU 1 and 2 threaded test

I’d say these are equivalent, which probably shows how little overhead there is from the hypervisor these days, though nested virtualization is a bit slower.

CPU 4 threaded test

Good to see that 2 cores actually performs faster than 1 core on a 4 threaded test. Nothing to see here…

Memory Operations Per Second

Horrible performance with nested, but with the hypervisor on bare metal ESXi and bhyve performed identically.

Memory MB/s

Once again nested virtualization was slow.. other than that neck and neck performance.

OLTP Transactions Per Second

The ESXi environment clearly takes the lead over bhyve, especially as the number of cores / threads started increasing. This is interesting because ESXi outperforms despite an I/O penalty from using NFS so ESXi is more than making up for that somewhere else.

Disk I/O Requests per Second

Clearly there’s an advantage to using local ZFS storage vs NFS. I’m a bit disappointing in the nested virtualization performance since from a storage standpoint it should be equivalent to bare metal FreeNAS, but may be due to the slow memory performance in that environment.

Disk Sequential Read/Write MBps

No surprises, ZFS local storage is going to outperform NFS

Well there you have it. I think it’s safe to say that bhyve is a viable solution for home (although I would like to see more people using it in the wild before considering it robust–I imagine we’ll see more of that now that FreeNAS has a UI for it). For low resource VMs E2 (nested virtualization) is a way to migrate between E1 and E3–but it’s not going to work for high performance VMs because of the memory performance hit.

This guide will install FreeNAS 10 (Corral) under VMware 6.5 ESXi, then via NFS share ZFS backed storage back to VMware. This is an update of my FreeNAS 9.10 on VMware 6.0 Guide.

“Hyperconverged” Design Overview

FreeNAS is installed as a Virtual Machine on the VMware Hypervisor. An LSI HBA in IT Mode is passed to FreeNAS via VT-d Passthrough. A ZFS pool is created on the disks attacked to the HBA. ZFS provides RAID-Z redundancy and an NFS dataset is then shared from FreeNAS and mounted from VMware which is used to provide storage for the remaining guests. Optionally containers and VM guests can run directly on FreeNAS itself using bhyve.

FreeNAS Corral

FreeNAS 10 (now called FreeNAS Corral) is a major rewrite over FreeNAS 9.10, the GUI has been overhauled, it has a CLI interface, and an API. I think the best feature is the bhyve hypervisor and docker support. To some degree for a single all-in-one hypervisor+NAS server you may not even need VMware and be able to get away with bhyve and docker.

Like anything new I advise caution against running it in a production environment. I do see quite a few rough edges and a few missing features that are available in FreeNAS 9.10. I imagine we’ll see frequent updates with polishing and features added. A good rule of thumb is to wait until TrueNAS hardware is shipping with the “Corral” version. I think this is the best release of FreeNAS yet, and it is going to be a great platform moving forward!

1. Get Hardware

This is based on my Supermicro X10SDV Build. For drives I used 4 x White Label NAS class HDDs (see ZFS Hard Drive Guide) and two Intel DC S3700s (similar models between S3500 and S3720 should be fine), which often show up for a decent price on Ebay. One SSD will be used to boot VMware and provide the initial data storage and the other used as a ZIL.

2. IPMI Setup

Go ahead and plug in the network cables to the IPMI management port, as well as at least one of the normal ethernet ports.

This should work with just about any server class Supermicro board…. first download the Supermicro IPMIView tool (I just enter “Private” for the company). Once installed run “IPMIView20” from the Start Menu (you may need to run it as Administrator).

Scan for IPMI Devices… once it finds your Supermicro server select it and Save.

Login to IPMI using ADMIN / ADMIN (you’ll want to change that obviously).

4. PCI Passthrough HBA

5. Setup VMware Storage Network

In the examples below my LAN / VM Network is on 10.2.0.0/16 (255.255.0.0) and my Storage network is on 10.55.0.0/16. You may need to adjust for your network. My storage network is on VLAN 55.

I like to keep my Storage Network separate from my LAN / VM Network. So we’ll create a VM Storage Network portgroup with a VLAN ID of 55.

Networking, Port groups, Add Port Group

Add VM Storage Network with VLAN ID of 55.

(you can choose a different VLAN ID, my storage network is 10.55.0.0/16 so I use “55” to match the network so that I don’t have to remember what VLAN goes to what network, but it doesn’t have to match).

Add a second port group just like it called Storage Network with the same VLAN ID (55).

Add VMKernel NIC

Attach it to the Storage Network and give it an address of 10.55.0.4 with a netmask of 255.255.0.0

You should end up with this…

6. Create a FreeNAS Corral VM

FreeBSD (64-bit)

Install it to the DC S3700 Datastore that VMware is installed on.

Add PCI Device and Select your LSI Card.

Add a second NIC for the VM Storage Network. You should have two NICS for FreeNAS, a VM Network and a VM Storage Network and you should set the Adapter Type to VMXNET 3 on both.

I usually give my FreeNAS VM 2 cores, if doing anything heavy (especially if you’ll be running docker images or bhyve under it you may want to increase that count). One rule with VMware is do not give VMs more cores than they need. I usually give each VM one core and only consider more if that particular VM needs more resources. This will reduce the risk of CPU co-stops from occurring. Gabrie van zanten’s How too many vCPUs can negatively affect performance is a good read.

ZFS needs memory. FreeNAS 10 needs 8GB memory minimum. Lock it.

Made the Hard Disk VMDK 16GB. There’s an issue with the VMware 6.5 SCSI controller on FreeBSD/FreeNAS. You’ll know it if you see an error like:

To prevent this, change the Virtual Device Node on the hard drive to SATA controller 0, and SCSI Controller 0 should be LSI Logic SAS

Add CD/DVD Drive, under CD/DVD Media hit Browse to upload and select the FreeNAS Corral ISO file which you can download from FreeNAS.

7. Install FreeNAS VM

Power on the VM…

Select the VMware disk to install to. I should note that if you create two VMDKs you can select them both at this screen and it will create a ZFS boot mirror, if you have an extra hard drive you can create another VMware data store there and put the 2nd vmdk there. This would provide some extra redundancy for the FreeNAS boot pool. In my case I know the DC S3700s are extremely reliable, and if I lost the FreeNAS OS I could just re-import the pool or failover to my secondary FreeNAS server.

Boot via BIOS.

Once FreeNAS is installed reboot and you should get the IP from DHCP on the console (once again I suggest setting this to a static IP).

If you hit that IP with a browser you should have a login screen!

8. Update and Reboot

Before doing anything…. System, Updates, Update and Reboot.

(Note: to get better insight into a task progress head over to the Console and type: task show).

9. Setup SSL Certificate

First, set your hostname, and also create a DNS entry pointing at the FreeNAS IP.

Create Internal CA

Export Certificate

Untar the file and click the HobbitonCA.crt to install it, install it to the trusted Root Certificate Authorities. I should note that if someone were to compromise your CA or gain the key they could do a MITM attack on you forging SSL certificates for other sites.

Create a Certificate for FreeNAS

Listen on HTTP+HTTPS and select the Certificate. I also increase the token Lifetime since I religiously lock my workstation when I’m away.

And now SSL is Secured

10. Create Pool

Do you want Performance, Capacity, or Redundancy? Drag the white circle thing where you want on the triangle and FreeNAS will suggest a zpool layout. With 4 disks I chose “Optimal” and it suggested RAID-Z which is what I wanted. Be sure to add the other SSD as a SLOG / ZIL / LOG.

11. Create Users

It’s probably best not to be logging in as root all the time. Create some named users with Administrator access.

12. Create Top Level Dataset

I like to create a top level dataset with a unique name for each FreeNAS server, that way it’s easier to replicate datasets to my other FreeNAS servers and perform recursive tasks (such as snapshots, or replication) on that top level dataset without having to micromanage them. I know you can sometimes do recursive tasks on the entire pool, but oftentimes I want to exclude certain datasets from those tasks (such as if those datasets are being replicated from another server).

Go to another menu option and then back to Storage, tank3, Replications, replication_ds4, and Start the replication and check back in a couple hours to make sure it’s working. My first replication attempt hung, so I canceled the task and started it again. I also found that adjusting the peer interval from 1 minute to 5 seconds under Peering may have helped.

16.1 Offsite Backups

It’s also a good idea to have Offsite backups, you could use S3, or a CrashPlan Docker Container, etc.

17. Setup Notifications

You want to be notified when something fails. FreeNAS can be configured to send an email or sent out Pushbullet notifications. Here’s how to setup Pushbullet.

The A.R.M. (Automatic Ripping Machine) detects the insertion of an optical disc, identifies the type of media and autonomously performs the appropriate action:

DVD / Blu-ray -> Rip with MakeMKV and Transcode with Handbrake

Audio CD -> Rip and Encode to FLAC and Tag the files if possible.

Data Disc -> Make an ISO backup

It runs on Linux, it’s completely headless and fully automatic requiring no interaction or manual input to complete it’s tasks (other than inserting the disk). Once it completes a rip it ejects the disc for you and you can pop in another one.

Automatic Ripping Machine Features

Determines if disc is Video, Data, or Audio

If video get the Title

Determine if it’s a TV or Movie

Rip using MakeMKV

Send rip to Handbrake and eject disc asynchronously

When done transcoding tell Emby to rescan library, or send notifications using PushBullet or IFTT

If audio CD – rip to mp3 or flac using abcde and eject

If data disc make an ISO backup

Can rip from multiple optical drives simultaneously

Completely headless design–no graphical interface. The only interaction is inserting the disc and it takes it from there, ejecting it when done.

Can send push notifications to your phone using Pushbullet or IFTT.

Free Software

I uploaded the scripts to GitHub under the MIT license. Since then quite a few contributors have made many improvements. Now the ARM has been rewritten in Python and has a number of options to customize the way different people handle media.

Instructions to get it installed on Ubuntu 16.04 or 18.04 LTS follows.

ARM Equipment & Hardware

Blu-Ray Hardware and VMware Settings

A WARNING ABOUT SOME BLU-RAY DRIVES

Most Blu-Ray drives have an anti-feature called “riplock” where it will purposefully cripple the read-speed on dvds and blue-rays to around 2X to 4X instead of the advertised drive speed (I believe this to be false advertising). If you have a normal 5 1/4″ drive bay I suggest getting the LG WH16NS40 16X blu-ray drive since it is known to not be speed limited. LG seems to be one of the better drive manufacturers in my experience.

You will need a server. You can use Ubuntu on bare metal or run it under VMware. I am using my Datacenter in a Box Build and run the ARM on Ubuntu Linux 16.04 LTS under VMware. At first I tried using an external USB Blu-Ray drive but the VM didn’t seem to be able to get direct access to it. Unfortunately my server case only has a slim-DVD slot on it so I purchased the Panasonic UJ160 Blu-Ray Player Drive because it was one of the cheaper Blu-Ray drives.

I wasn’t sure if VMware would recognize the Blu-Ray functions on the drive but it does! Once physically installed edit the VM properties so that it uses the host device as the CD/DVD drive and then select the optical drive.

After doing a little research I found out DVD and Blu-Ray players have region codes that only allow them to play movies in the region they were intended–by default the Panosonic drive shipped with a region code set to 0.

Looking at http://tdb.rpc1.org/ it looks like it is possible to flash some drives so that they can play videos in all region codes. Fortunately before I got too far down the flash the drive path I discovered you can simply change the region code! Since I’m only playing North American movies I set the region code to 1 using:

1

2

sudo apt-getinstall regionset

sudo regionset/dev/sr0

You can only change this setting 4 or 5 times then it gets stuck so if you’re apt to watch movies from multiple regions you’ll want to look at getting a drive that you can flash the firmware.

Once that’s in the file mount the folder and create an ARM and an ARM/raw folder.

1

2

3

sudo mkdir/mnt/media

sudo mount/mnt/media

sudo mkdir-p/mnt/media/ARM/raw

Configure ARM Settings

1

You should look over the config file to make sure it suits your needs, if you want to add Android or iOS push notifications, trans-coding options, subtitle options, etc. that’s where to do it.

Figure out how to restart udev, or reboot the VM (make sure your media folder gets mounted on reboot). You should be set.

Automatic Ripping Machine Usage

Insert Disc.

Wait until the A.R.M. ejects the disc.

Repeat

Test out a movie, audio cd, and data cd and make sure it’s working as expected. Check the logs. If you run into trouble feel free to post an issue here.

Install MakeMKV License

MakeMKV will run on a trial basis for 30 days. Once it expires you’ll need to purchase a key or while it’s in BETA you can get a free key… I would love to build this solution on 100% free open source software but MakeMKV saves so much time and is more reliable compared to anything else I’ve tried. I ended up purchasing a license key for myself.

Get an OMDB API Key

Next you’ll want to get an OMDB API key and put it in your ARM config file. A free key will let you do 1,000 API queries a day which should be more than enough: http://www.omdbapi.com/apikey.aspx

How it Works?

When UDEV/systemd detects a disc insert as defined by /lib/udev/rules.d/51-automedia.rules it runs the wrapper which in turn runs /opt/arm/identify.sh which identifies the type of media inserted and then calls the appropriate scripts. (if you ever need it this is a great command get get info on a disk):

1

udevadm info-qenv-n/dev/sr0

Video Discs (Blu-Ray/DVD)

For video discs the first step is ARM tries to obtain the disc title. If it’s a blu-ray it can often be extracted from the disc, if it’s a DVD we calculate a hash of the DVD and then query Windows Media Metaservice (which is what Windows Media Player queries when a disc is inserted) to get the title.

Once the title is obtained we send that to the OMDB API which will tell us whether the video is a Movie, or a TV Show. If the video is a Movie ARM can usually determine the main title feature, and rip that. And optionally rip all the other titles into an Extras folder. Once done ARM can automatically tell Emby to rescan the library. If the video is a TV Show ARM will rip all the titles and you’ll need to use Filebot to rename the shows.

All tracks get ripped using MakeMKV and placed in the /mnt/media/ARM/raw folder as soon as ripping is complete the disk ejects and transcoding starts with HandBrakeCli transcoding every track into /mnt/media/ARM/timestamp_discname. You don’t have to wait for transcoding to complete, you can immediately insert the next disk to get it started.

Most of the time everything just works, but in some cases if ARM can’t determine the title some video file renaming needs to be done by hand. The ARM will name the folder using the disc title, but this isn’t always accurate. For a Season of TV shows I’ll name them using FileBot and then move them to one of the Movie or TV folders that my Emby Server looks at. Fortunately this manual part of the process can be done at any time, it won’t hold up ripping more media. The Emby Server then downloads artwork and metadata for the videos.

Audio CDs

If an audio track is detected it is ripped to a FLAC file (or mp3 or whatever you want) using the abcde ripper. I opted for the FLAC format because it’s lossless, well supported, and is un-proprietary. If you’d prefer a different format ABCDE can be configured to rip to MP3, AAC, OGG, whatever you want. I have it dropping the audio files in the same location as the video files but I could probably just move it directly to the music folder where Emby is looking.

Data Disks (Software, Pictures, etc.)

If ARM determines there is no video on the disc, then a simple script is run to make a backup ISO image of the disc.

Morality of Ripping

Two Evils: Piracy vs. DRM

I am for neither Piracy or DRM. Where I stand morally is I make sure we own every CD, DVD, and Blu-Ray that we rip using the ARM.

I don’t advocate piracy. It is immoral for people to make copies of movies and audio they don’t own. On the other hand there is a difference between Piracy and copying for fair use which publisher’s often wrongly lump together.

For me it doesn’t really even matter because I don’t really like watching movies anyway–there’s not much more painful than sitting for an hour to get through a movie. I just like making automatic ripping machines.

Playing with bhyve

Here’s a look at Gea’s popular All-in-one design which allows VMware to run on top of ZFS on a single box using a virtual 10Gbe storage network. The design requires an HBA, and a CPU that supports VT-d so that the storage can be passed directly to a guest VM running a ZFS server (such as OmniOS or FreeNAS). Then a virtual storage network is used to share the storage back to VMware.

VMware and ZFS: All-In-One Design

bhyve, can simplify this design since it runs under FreeBSD it already has a ZFS server. This not only simplifies the design, but it could potentially allow a hypervisor to run on simpler less expensive hardware. The same design in bhyve eliminates the need to use a dedicated HBA and a CPU that supports VT-d.

Simpler bhyve design

I’ve never understood the advantage of type-1 hypervisors (such as VMware and Xen) over Type-2 hypervisors (like KVM and bhyve). Type-1 proponents say the hypervisor runs on bare metal instead of an OS… I’m not sure how VMware isn’t considered an OS except that it is a purpose-built OS and probably smaller. It seems you could take a Linux distribution running KVM and take away features until at some point it becomes a Type-1 hypervisor. Which is all fine but it could actually be a disadvantage if you wanted some of those features (like ZFS). A type-2 hypervisor that supports ZFS appears to have a clear advantage (at least theoretically) over a type-1 for this type of setup.

In fact, FreeBSD may be the best visualization / storage platform. You get ZFS and bhyve, and also jails. You really only need to run bhyve when virtualizing a different OS.

bhyve is still pretty young, but I thought I’d run some tests to see where it’s at…

3 – VM under FreeBSD bhyve

4 – VM under FreeBSD bhyve sync always

bhyve running on FreeBSD 10.1-Release

Guest storage is file image on ZFS dataset.

Sync=always

Benchmark Results

MariaDB OLTP Load

This test is a mix of CPU and storage I/O. bhyve (yellow) pulls ahead in the 2 threaded test, probably because it doesn’t have to issue a sync after each write. However, it falls behind on the 4 threaded test even with that advantage, probably because it isn’t as efficient at handling CPU processing as VMware (see next chart on finding primes).

Finding Primes

Finding prime numbers with a VM under VMware is significantly faster than under bhyve.

Random Read

byhve has an advantage, probably because it has direct access to ZFS.

Random Write

With sync=standard bhyve has a clear advantage. I’m not sure why VMware can outperform bhyve sync=always. I am merely speculating but I wonder if VMware over NFS is translating smaller writes into larger blocks (maybe 64k or 128k) before sending them to the NFS server.

Random Read/Write

Sequential Read

Sequential reads are faster with bhyve’s direct storage access.

Sequential Write

What not having to sync every write will gain you..

Sequential Rewrite

Summary

VMware is a very fine virtualization platform that’s been well tuned. All that overhead of VT-d, virtual 10gbe switches for the storage network, VM storage over NFS, etc. are not hurting it’s performance except perhaps on sequential reads.

For as young as bhyve is I’m happy with the performance compared to VMware, it appears to be a slower on the CPU intensive tests. I didn’t intend on comparing CPU performance so I haven’t done enough variety of tests to see what the difference is there but it appears VMware has an advantage.

One thing that is not clear to me is how safe running sync=standard is on bhyve. The ideal scenario would be honoring fsync requests from the guest, however I’m not sure if bhyve has that kind of insight from the guest. Probably the worst case under this scenario with sync=standard is losing the last 5 seconds of writes–but even that risk can be mitigated with battery backup. With standard sync there’s a lot of performance to be gained over VMware with NFS. Even if you run bhyve with sync=always it does not perform badly, and even outperforms VMware All-in-one design on some tests.

The upcoming FreeNAS 10 may be an interesting hypervisor + storage platform, especially if it provides a GUI to manage bhyve.

This is a guide which will install FreeNAS 9.10 under VMware ESXi and then using ZFS share the storage back to VMware. This is roughly based on Napp-It’s All-In-One design, except that it uses FreeNAS instead of OminOS.

Disclaimer: I should note that FreeNAS does not officially support running virtualized in production environments. If you run into any problems and ask for help on the FreeNAS forums, I have no doubt that Cyberjock will respond with “So, you want to lose all your data?” So, with that disclaimer aside let’s get going:

Update: Josh Paetzel wrote a post onVirtualizing FreeNASso this is somewhat “official” now. I would still exercise caution.

Update 2: This guide was originally written for FreeNAS 9.3, I’ve updated it for FreeNAS 9.10. Also, I believe Avago LSI P20 firmware bugs have been fixed and have been around long enough to be considered stable so I’ve removed my warning on using P20. Added sections 7.1 (Resource reservations) and 16.1 (zpool layouts) and some other minor updates.

The LSI2308/M1015 has 8 ports, I like do to two DC S3700s for a striped SLOG device and then do a RAID-Z2 of spinners on the other 6 slots. Also get one (preferably two for a mirror) drives that you will plug into the SATA ports (not on the LSI controller) for the local ESXi data store. I’m using DC S3700s because that’s what I have, but this doesn’t need to be fast storage, it’s just to put FreeNAS on.

2. Flash HBA to IT Firmware

As of FreeNAS 9.3.1 or greater you should be flashing to IT mode P20 (looks like it’s P21 now but it’s not available by every vendor yet).

(If you already have the card passed through to FreeNAS via VT-d (steps 6-8) you can actually flash the card from FreeNAS using the sas2flash utility using the steps below (in this example my card is already in IT mode so I’m just upgrading it):

(Wait a few minutes, at this point FreeNAS finally crashed. Poweroff. FreeNAS, and then reboot VMware)

Warning on P20 buggy firmware:

Some earlier versions of the P20 firmware were buggy, so make sure it’s version P20.00.04.00 or later. If you can’t P20 in aversion later than P20.00.04.00 then use P19 or P16.

3. Optional: Over-provision ZIL / SLOG SSDs.

If you’re going to use an SSD for SLOG you can over-provision them. You can boot into an Ubuntu LiveCD and use hdparm, instructions are here: https://www.thomas-krenn.com/en/wiki/SSD_Over-provisioning_using_hdparm You can also do this after after VMware is installed by passing the LSI controller to an Ubuntu VM (FreeNAS doesn’t have hdparm). I usually over-provision down to 8GB.

Update 2016-08-10: But you may want to only go to 20GB depending on your setup! One of my colleagues discovered 8GB over-provisioning wasn’t even maxing out 10Gb network (remember, every write to VMware is a sync so it hits the ZIL no matter what) with 2 x 10Gb fiber lagged connections between VMware and FreeNAS. This was on an HGST 840z so not sure if the same holds true for the Intel DC S3700… and it wasn’t virtualized setup. But thought I’d mention it here.

4. Install VMware ESXi 6

Under configuration, storage, click add storage. Choose one (or two) of the local storage disks plugged into your SATA ports (do not add a disk on your LSI controller).

5. Create a Virtual Storage Network.

For this example my VMware management IP is 10.2.0.231, the VMware Storage Network ip is 10.55.0.2, and the FreeNAS Storage Network IP is 10.55.1.2.

Create a virtual storage network with jumbo frames enabled.

VMware, Configuration, Add Networking. Virtual Machine…

Create a standard switch (uncheck any physical adapters).

Add Networking again, VMKernel, VMKernel… Select vSwitch1 (which you just created in the previous step), give it a network different than your main network. I use 10.55.0.0/16 for my storage so you’d put 10.55.0.2 for the IP and 255.255.0.0 for the netmask.

Some people are having trouble with an MTU of 9000. I suggest leaving the MTU at 1500 and make sure everything works there before testing an MTU of 9000. Also, if you run into networking issues look at disabling TSO offloading (see comments).

Under vSwitch1 go to Properties, select vSwitch, Edit, change the MTU to 9000. Answer yes to the no active NICs warning.

Then select the Storage Kernel port, edit, and set the MTU to 9000.

6. Configure the LSI 2308 for Passthrough (VT-d).

Configuration, Advanced Settings, Configure Passthrough.

Mark the LSI2308 controller for passthrough.

You must have VT-d enabled in the BIOS for this to work so if it won’t let you for some reason check your BIOS settings.

Reboot VMware.

7. Create the FreeNAS VM.

Create a new VM, choose custom, put it on one of the drives on the SATA ports, Virtual Machine version 11, Guest OS type is FreeBSD 64-bit, 1 socket and 2 cores. Try to give it at least 8GB of memory. On Networking give it two adapters, the 1st NIC should be assigned to the VM Network, 2nd NIC to the Storage network. Set both to VMXNET3.

SCSI controller should be the default, LSI Logic Parallel.

Choose Edit the Virtual Machine before completion.

If you have a second local drive (not one that you’ll use for your zpool) here you can add a second boot drive for a mirror.

Before finishing the creation of the VM click Add, select PCI Devices, and choose the LSI 2308.

And be sure to go into the CD/DVD drive settings and set it to boot off the FreeNAS iso. Then finish creation of the VM.

7.1 FreeNAS VM Resource allocation

Also, since FreeNAS will be driving the storage for the rest of VMware, it’s a good idea to make sure it has a higher priority for CPU and Memory than other guests. Edit the virtual machine, under Resources set the CPU Shares to “High” to give FreeNAS a higher priority, then under Memory allocation lock the guest memory so that VMware doesn’t ever borrow from it for memory ballooning. You don’t want VMware to swap out ZFS’s ARC (memory read cache).

8. Install FreeNAS.

Boot of the VM, install it to your SATA drive (or two of them to mirror boot).

After it’s finished installing reboot.

9. Install VMware Tools.

SKIP THIS STEP. As of FreeNAS 9.10.1 installing VMware should may no longer be necessary–you can skip step 9 and go to 10. Just leaving this for historical purposes.

Once installed Navigate to the WebGUI, it starts out presenting a wizard, I usually set my language and timezone then exit the rest of the wizard.

Under System, Tunables… Add a Tunable. Variables should be: vmxnet3_load. The type should be Loader and the Value YES .

Reboot FreeNAS. On reboot you should notice that the VMXNET3 NICS now work (except the NIC on the storage network can’t find a DHCP server, but we’ll set it to static later), also you should notice that VMware is now reporting that VMware tools are installed.

If all looks well shutdown FreeNAS (you can now choose Shutdown Guest from VMware to safely power it off), remove the E1000 NIC and boot it back up (note that the IP address on the web gui will be different).

10. Update FreeNAS

Before doing anything let’s upgrade FreeNAS to the latest stable under System Update.

This is a great time to make some tea.

Once that’s done it should reboot. Then I always go back again and check for updates again to make sure there’s nothing left.

11. SSL Certificate on the Management Interface (optional)

On my DHCP server I’ll give FreeNAS a static/reserved IP, and setup an entry for it on my local DNS server. So for this example I’ll have a DNS entry on my internal network for stor1.b3n.org.

If you don’t have your own internal Certificate Authority you can create one right in FreeNAS:

System, CAs, Create internal CA. Increase the key length to 4096 and make sure the Digest Algorithm is set to SHA256.

Click on the CA you just created, hit the Export Certificate button, click on it to install the Root certificate you just created on your computer. You can either install it just for your profile or for the local machine, I usually do local machine, and you’ll want to make sure to store it is in the Trusted Root Certificate Authorities store.

Just a warning, that you must keep this Root CA guarded, if a hacker were to access this he could generate certificates to impersonate anyone (including your bank) to initiate a MITM attack.

Also Export the Private Key of the CA and store it some place safe.

Now create the certificate…

System, Certificates, CreateInternalCertificate. Once again bump the key length to 4096. The important part here is the Common Name must match your DNS entry. If you are going to access FreeNAS via IP then you should put the IP address in the Common Name field.

System, Information. Set the hostname to your dns name.

System, General. Change the protocol to HTTPS and select the certificate you created. Now you should be able to go to use https to access the FreeNAS WebGUI.

12. Setup Email Notifications

Account, Users, Root, Change Email, set to the email address you want to receive alerts (like if a drive fails or there’s an update available).

System, Advanced

Show console messages in the footer. Enable (I find it useful)

System Email…

Fill in your SMTP server info… and send a test email to make sure it works.

13. Setup a Proper Swap

FreeNAS by default creates a swap partition on each drive, and then stripes the swap across them so that if any one drive fails there’s a chance your system will crash. We don’t want this.

System, Advanced…

Swap size on each drive in GiB, affects new disks only. Setting this to 0 disables swap creation completely (STRONGLY DISCOURAGED). Set this to 0.

Open the shell. This will create a 4GB swap file (based on https://www.freebsd.org/doc/handbook/adding-swap-space.html)

Next time you reboot on the left Navigation pane click Display System Processes and make sure the swap shows up. If so it’s working.

14. Configure FreeNAS Networking

Setup the Management Network (which you are currently using to connect to the WebGUI).

Network, Interfaces, Add Interface, choose the Management NIC, vmx3f0, and set to DHCP.

Setup the Storage Network

Add Interface, choose the Storage NIC, vmx3f1, and set to 10.55.1.2 (I setup my VMware hosts on 10.55.0.x and ZFS servers on 10.55.1.x), be sure to select /16 for the netmask. And set the mtu to 9000.

Open a shell and make sure you can ping the ESXi host at 10.55.0.2

Reboot. Let’s make sure the networking and swap stick.

15. Hard Drive Identification Setup

Label Drives. FreeNAS is great at detecting bad drives, but it’s not so great at telling you which physical drive is having an issue. It will tell you the serial number and that’s about it. But how confident are you in knowing which drive fails? If FreeNAS tells you that disk da3 (by the way, all these da numbers can change randomly) is having an issue how do you know which drive to pull? Under Storage, View Disks, you can see the serial number, this still isn’t entirely helpful because chances are you can’t see the serial number without pulling a drive. So we need to map them to slot numbers or labels of some sort.

There are two ways you can deal with this. The first, and my preference, is sas2ircu. Assuming you connected the cables between the LSI 2308 and the backplane in proper sequence sas2ircu will tell you the slot number the drives are plugged into on the LSI controller. Also if you’re using a backplane with an expander that supports SES2 it should also tell you which slots the drives are in. Try running this command:

1

# sas2ircu 0 display|less

You can see that it tells you the slot number and maps it to the serial number. If you are comfortable that you know which physical drive each slot number is in then you should be okay.

If not, the second method, is remove all the drives from the LSI controller, and put in just the first drive and label it Slot 0 in the GUI by clicking on the drive, Edit, and enter a Description.

Put in the next drive in Slot 1 and label it, then insert the next drive and label it Slot 2 and so on…

The Description will show up in FreeNAS and it will survive reboots. it will also follow the drive even if you move it to a different slot. So it may be more appropriate to make your description match a label on the removable trays rather than the bay number.

It doesn’t matter if you label the drives or use sas2ircu, just make sure you’re confident that you can map a serial number to a physical drive before going forward.

16.1 Choose Pool Layout

For high performance the best configuration is to maximize the number of VDEVs by creating mirrors (essentially RAID-10). That said, with my 6-drive RAID-Z2 array with 2 DC S3700 SSDs for SLOG/ZIL my setup performs very well with VMware in my environment. If you’re running heavy random I/O mirrors are more important, but if you’re just running a handful of VMs RAID-Z / RAID-Z2 will probably offer great performance as long as you have a good SSD for SLOG device. I like to start double parity at 5 or 6 disk VDEVs, and triple parity at 9 disks. Here some some sample configurations:

Anyway, that gives you a rough idea. The more vdevs the better random performance. It’s always a balance between capacity, performance, and safety.

16.2 Create the Pool.

Storage, Volumes, Volume Manager.

Click the + next to your HDDs and add them to the pool as RAID-Z2.

Click the + next to the SSDs and add them to the pool. By default the SSDs will be on one row and two columns. This will create a mirror. If you want a stripe just add one Log device now and add the second one later. Make certain that you change the dropdown on the SSD to “Log (ZIL)” …it seems to lose this setting anytime you make any other changes so change that setting last. If you do not do this you will stripe the SSD with the HDDs and possibly create a situation where any one drive failure can result in data loss.

Back to Volume manager and add the second Log device…

I have on numerous occasions had the Log get changed to Stripe after I set it to Log, so just double-check by clicking on the top level tank, then the volume status icon and make sure it looks like this:

17. Create an NFS Share for VMware

You can create either an NFS share, or iSCSI share (or both) for VMware. First here’s how to setup an NFS share:

In VMware, Configuration, Add Storage, Network File System and add the storage:

And there’s your storage!

18. Create an iSCSI share for VMware

WARNING: Note that at this time, based on some of the comments below with people having connection drop issues on iSCSI I suggest testing with heavy concurrent loads to make sure it’s stable. Watch dmesg and /var/log/messages on FreeNAS for iSCSI timeouts. Personally I use NFS. But here’s how to enable iSCSI:

Storage, select the nested tank, Create Zvol. Be sure compression is set to lz4. Check Sparse Volume. Choose advanced mode and optionally change the default block size. I use 64K block-size based on some benchmarks I’ve done comparing 16K (the default), 64K, and 128K. 64K blocks didn’t really hurt random I/O but helped some on sequential performance, and also gives a better compression ratio. 128K blocks had the best better compression ratio but random I/O started to suffer so I think 64K is a good middle-ground. Various workloads will probably benefit from different block sizes.

Sharing, Block (iSCSI), Target Global Configuration.

Set the base name to something sensible like: iqn.2011-03.org.b3n.stor1.istgt Set Pool Available Space Threshold to 60%

20. Periodic Snapshots

Add periodic snapshot jobs for your VMware storage under Storage, Periodic Snapshot Tasks. You can setup different snapshot jobs with different retention policies.

21. ZFS Replication

If you have a second FreeNAS Server (say stor2.b3n.org) you can replicate the snapshots over to it. On stor1.b3n.org, Replication tasks, view public key. copy the key to the clipboard.

On the server you’re replicating to, stor2.b3n.org, go to Account, View Users, root, Modify User, and paste the public key into the SSH public Key field. Also create a dataset called “replicated”.

Back on stor1.b3n.org:

Add Replication. Do an SSH keyscan.

And repeat for any other datasets. Optionally you could also just replicate the entire pool with the recursive option.

22. Automatic Shutdown on UPS Battery Failure (Work in Progress)

The goal is on power loss, before the battery fails to shutdown all the VMware guests including FreeNAS. So far all I have gotten is the APC working with VMware. Edit the VM settings and add a USB controller, then add a USB device and select the UPS, in my case a APC Back-UPS ES 550G. Power FreeNAS back on.

On the shell type:

dmesg|grep APC

1

<span class="shell_ff shell_be">ugen0.4:&lt;APC&gt;at usbus0</span>

This will tell you where the APC device is. IN my case it’s showing up on ugen0.4. I ended up having to grant world access to the UPS…

Shell

1

chmod777/dev/ugen0.4

For some reason I could not get the GUI to connect to the UPS, I can selected ugen0.4, but under the drivers dropdown I just have hyphens —— … I set it manually in /usr/local/etc/nut/ups.conf

1

2

3

4

[ups]

driver=usbhid-ups

port=/dev/ugen0.4

desc="APC 1"

However, this file gets overwritten on reboot, and also the rc.conf setting doesn’t seem to stick. I added this tunable to get the rc.conf setting…

And I created my ups.conf file in /mnt/tank/ups.conf. Then I created a script to stop the nut service, copy my config file and restart the nut service in /mnt/tank/nutscript.sh

Shell

1

2

3

4

#!/bin/sh

service nut stop

cp/mnt/tank/ups.conf/usr/local/etc/nut/ups.conf

service nut start

Then under tasks, Init/Shutdown Scripts I added a task to run the script post init.

Next step is to configure automatic shutdown of the VMware server and all guests on it… I have not done this yet.

There’s a couple of approaches to take here. One is to install a NUT client on the ESXi, and the other is to have FreeNAS ssh into VMware and tell it to shutdown. I may update this section later if I ever get around to implementing it.

23. Backups

Before going live make sure you have adequate backups! You can use ZFS replication with a fast link. For slow network connections Rsync will work better (Took under Tasks -> Rsync tasks) or use a cloud service like CrashPlan. Here’s a nice CrashPlan on FreeNAS Howto.

BACKUPS BEFORE PRODUCTION. I can’t stress this enough, don’t rely on ZFS’s redundancy alone, always have backups (one offsite, one onsite) in place before putting anything important on it.

. <-- this is a dot

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 158 other subscribers

Email Address

b3n.org is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com