Here’s a quick guide to run an NTP (Network Time Protocol) server using Chrony with a GPS (optional) receiver on a VMware ESXi Guest running Ubuntu 18.04. I should note this is experimental and something I setup in my homelab temporarily. For production environments I would run NTP on physical hardware and not VMware.

Create and Configure VM

Be sure to disable Guest Tools Time synchronization by editing the VM settings and uncheck Synchronize guest time with host.

Set the CPU shares to High… we want the NTP server to have priority if there is processor contention.

Install Chrony

Shell

1

2

sudo apt install chrony

sudo vim/etc/chrony/chrony.conf

/etc/chrony/chrony.conf

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

# Welcome to the chrony configuration file. See chrony.conf(5) for more

# information about usuable directives.

# This will use (up to):

# 3 sources from ntp.ubuntu.com

# 3 sources from us.pool.ntp.org

# 3 sources from time.nist.gov

pool ntp.ubuntu.comiburst maxsources3

pool us.pool.ntp.orgiburst maxsources3

pool time.nist.goviburst maxsources3

# This directive specify the location of the file containing ID/key pairs for

# NTP authentication.

keyfile/etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate

# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.

rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than

# one second

makestep1-1

allow

I diversified between Ubuntu’s, NTP.org’s and NIST’s time server pools.

That’s it, after restarting the chrony service (service restart chrony) you should be able to get time reports by running:

Shell

1

2

3

chronyc sources-v

chronyc sourcestats-v

chronyc tracking-v

Why You Shouldn’t Run an NTP Server in a VM Guest

VM’s can’t keep accurate time

I’ve generally found that VMs keep great time inside of VMware. One thing that can help with this is setting the CPU shares to high so your time server always has a priority. I ran Chrony in a VM for several weeks, compared it with Chrony on a Raspberry Pi. Both were acceptable, and both had a smaller standard deviation than public NTP servers over the internet, but the VM had a much smaller standard deviation than the Pi. That tells me VMs running on better hardware may be better than lesser bare physical hardware at time tracking under certain conditions, and a local NTP server in a VM can be more precise than grabbing time off the internet.

VMs can become out of sync during snapshots, suspend, failover, etc.

I ran a suspend test and this is true. I paused a VM, waited 10 seconds, then resumed it. It reported the wrong time to NTP clients for several minutes before it corrected itself from external NTP servers. Here’s a screenshot of my NTP server being 11 seconds off after a pause!

This is a valid reason to run an NTP server on physical hardware. However, I think it is possible to run an NTP server under VMware with the following precautions:

Your NTP servers under VMware should never be paused. That means they should be excluded from failover (instead of failover it’s better to configure multiple NTP servers for your clients to connect to since it’s better for an NTP server to be down than report a wrong time).

Have multiple NTP servers. At least three. You’ll notice in the screenshot above Chrony (running on a separate physical machine) flagged the server as not being accurate. This way if one of your VMs gets paused chrony will switch to another time-source automatically.

Set makestep 1 -1 in the chrony.conf file (this tell chrony that any difference greater than one second will get stepped which allows for faster correction after a resume).

GPS Receiver

This is not really related to VMware. But I had a GPS receiver so thought I’d see how it works with Chrony….

I have a GlobalSat BU-353S4 USB GPS Receiver. This isn’t the best GPS receiver for accuracy. For me it’s accurate to within a few hundred milliseconds which is good enough for my experimental purposes but worse than just grabbing time off the internet. For serious time-keepers you’ll be wanting to use something faster than USB and more accurate than what a cheap GPS receiver can provide.

Configure gpsd

Shell

1

2

sudo apt install gpsd gpsd-clients

sudo vim/etc/default/gpsd

/etc/default/gpsd

Shell

1

2

3

4

5

6

7

8

9

10

11

12

# Start the gpsd daemon automatically at boot time

START_DAEMON="true"

# Use USB hotplugging to add new USB devices automatically to the daemon

USBAUTO="true"

# Devices gpsd should collect to at boot time.

# They need to be read/writeable, either by user gpsd or the group dialout.

DEVICES="/dev/ttyUSB0"

# Other options you want to pass to gpsd

GPSD_OPTIONS="-n -F /var/run/gpsd.sock"

Install Chrony

1

2

sudo apt install chrony

sudo vim/etc/chrony/chrony.conf

/etc/chrony/chrony.conf

Shell

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

# Welcome to the chrony configuration file. See chrony.conf(5) for more

# information about usuable directives.

# This will use (up to):

3sources from ntp.ubuntu.com

3sources from us.pool.ntp.org

3sources from time.nist.gov

1GPS device

pool ntp.ubuntu.comiburst maxsources3

pool us.pool.ntp.orgiburst maxsources3

pool time.nist.goviburst maxsources3

refclock SHM0refid GPS precision1e-1offset0.250delay1e-9

# This directive specify the location of the file containing ID/key pairs for

# NTP authentication.

keyfile/etc/chrony/chrony.keys

# This directive specify the file into which chronyd will store the rate

# real-time clock. Note that it can’t be used along with the 'rtcfile' directive.

rtcsync

# Step the system clock instead of slewing it if the adjustment is larger than

# one second, at any time

makestep1-1

allow

So, how did I get the values on the refclock line…

The way I came up with my offset of 0.250 is by initially setting the offset to 0.0, restarting chrony, and running chronyc sources -v several times taking note of the offset. I’d get numbers like +249ms, +253ms, +250ms, etc.

Since my GPS is off by about 250ms I set the offset to 0.250. Now it’s usually not off by more than 100ms.

The 100ms+- variance is not a problem when being combined with other sources, but if it was the only time source I’d be better off tolerating drift than the high variance of GPS for a short period without access to the NTP pools, if I had no internet for several months or an air-gapped network then time via GPS would probably be better than nothing–but a better GPS receiver should be used in those scenarios.

For most networks running chrony in a VM and using a GPS is unnecessary. It’s better to keep it simple. I just use the NTP service on my pfSense router and set all the clients to that.

There are plenty of books on managing people; but there are few books targeting management of software development, and even fewer aimed at people who got promoted into leadership positions with no management skills. I’ve read countless books looking for resources in that area… I can find plenty of books about how to manipulate people or promote yourself (and I’ve had plenty of training to that affect) but those are not the books I’m looking for.

I want real authentic leadership and practical management. Below you will find the best of what I’ve found over the last four years. And unlike some “Best Books for Programming Managers” and “Top 10 books on Leadership” lists you’ll find online… I actually read every book listed below.

I should also note that even if you aren’t in a position of management these books should be beneficial. Whether you have the position or not, everyone has the opportunity to lead.

Managing the Unmanageable

“Most successful programming managers are former programmers: They can quickly grasp whether a developer is on track through the most informal of conversations, without having to ferret out the assessment through long strings of questions that can feel pestering.”

Managing the Unmanageable is the comprehensive handbook to gain a variety of insights and a tool set to manage software development teams. I didn’t find it lacking coverage on any topic.

It rightly points out how managing programmers is like managing artists–programming is a creative job so you can’t manage that the same way you would manage most other jobs.

It goes over how to build relationships with and manage HR, your boss, other departments, etc. How to define developer levels, how not to do incentives (which can often be more demotivating than motivating), job descriptions, how to conduct interviews, build culture, motivate developers, etc. This is a wide book in what it covers. The vastness of topics is unmatched by any other management book I’ve read. It may only devote a few pages to some subjects but I haven’t found an area that it doesn’t cover at all. Even for areas it doesn’t go into great depth it references sources for further study.

I think this is the best resource for a new manager to get a comprehensive overview of every topic related to managing programmers. What I really like about the book is from the experience of the authors it anticipates and provides guidance on a lot of challenges I had to deal with–reading this book helped me proactively plan how to deal with those situations.

For me, reading Managing the Unmanageable is like sitting down at a coffee shop with some seasoned managers and listening to their experience and wisdom. Today I still use it as reference book.

Peopleware

“The major problems of our work are not so much technological as sociological in nature.”

“Most managers are willing to concede the idea that they’ve got more people worries than technical worries. But they seldom manage that way. They manage as though technology were their principal concern. They spend their time puzzling over the most convoluted and most interesting puzzles that their people will have to solve, almost as though they themselves were going to do the work rather than manage it.”

Peopleware, as it’s title suggests is all about the people aspect of managing software developers. It’s not a generic management book. Most of it only applies to managing creative and intellectual workers. It covers why programmers are distinct from and must be managed differently than other types of jobs, such as accountants or manufacturing workers. The book covers topics like the importance of allowing time to think on the job, giving teams a sense of elitism to increase productivity, creating environments where teams can naturally form and jell, the importance of an interruption free office environment, why the surest way to improve productivity is by focusing on quality.

I learned environmental factors for a programmer cause a 10 to 1 performance difference. A large section deals with the work environment. Office design, layouts, how bad cubicles are, the importance of natural light, office size, privacy, etc. This is a timeless classic. It would benefit any manager, executive, head of HR, architect, or programmer (even if you aren’t in a management position, this book will help you manage yourself).

The Mythical Man-Month

“Why is programming fun? What delights may its practitioner expect as his reward? First is the sheer joy of making things. As the child delights in his mud pie, so the adult enjoys building things, especially things by his own design. I think this delight must be an image of God’s delight in making things, a delight shown in the distinctness and newness of each leaf and each snowflake.”

This is a collection of essays about managing and organizing large software projects. Most important is Brooks’ observation that adding more man-power to a late software project will make it even later. My favorite observation of his was how the most productive teams are smaller because of the communication overhead, you only get fractional gains by increasing the size of large teams. Although pre-Agile, many of his ideas influenced Agile project management. He was well ahead of his time. This is a classic.

“Adding manpower to a late software project makes it later.”

The Conviction to Lead

“Whenever Christian leaders serve, in the church or in a secular world, their leadership should be driven by distinctively Christian conviction.”

“Leadership is all about putting the right beliefs into action, and knowing, on the basis of convictions, what those right beliefs and actions are. This book is written with the concern that far too much of what passes for leadership today is mere management. Without convictions you might be able to manage, but you cannot really lead.”

This was not an easy find. I read fluffy leadership book after fluffy leadership book… and finally read Mohler’s book at my dad’s recommendation. It has far more substance on leadership than anything else I’ve read. Where others give you mechanics, tools and methods, Mohler gives you conviction and motivation based on well grounded beliefs. It is not written just to pastors, nor just to leaders of Christian institutions (although this appears to be the main focus), but also to Christians who happen to be leaders in secular organizations–and that’s quite rare for a book on leadership written by a devout Christian.

Mohler’s book is practical because it provides the foundation for why and how Christians should be leading and the basis for leading in a secular world. I would say the book is primarily written to C-level, but almost all of it I was able to apply to a smaller realm for lower levels of management if I limited the scope to my area of influence. This is a good book for any Christian in a position of leadership.

I’ve been using a Kindle for about 6 years. And have been reading paper books for longer than that! I have two Kindles, one is the discontinued Kindle Touch, and the other is the newer Kindle Paperwhite. Here are my thoughts on the Kindle and how eBooks compare to Print Books.

The Kindle Reading Experience

For much of the reading experience I prefer the Kindle. It’s compact, lightweight, and easy to carry around. With a kindle I don’t have to awkwardly hold a book open while my other hand is trying to not spill my cup of coffee. Also when it starts to dim outside and I don’t quite have enough light I can turn on the backlight instead of the house lights.

Backlight

So, e-ink displays don’t have as good of a contrast as real paper. The reason Amazon calls their latest Kindle the “Paperwhite” is it has a backlight that can sort of match the brightness of paper by supplementing the light from your environment–the idea is you turn the backlight on just enough so that it still looks like it’s reflecting light like a book, but there’s just enough extra light to make it as readable as paper. This does work, however I think the LED color Amazon chose is a failure. The pure white LED backlight is too much in the blue spectrum and that’s very obvious when I’m reading under incandescent lights. It’s okay in natural light but under incandescent lighting it should be warmer to match the surrounding atmosphere. This could affect health if reading right before going to bed. I hope Amazon fixes this in the next version…maybe it should have RGB bulbs and a sensor to match the ambient light.

In very bright light paper wins out, but if the ambient light is dim as it often is in the Fall in Idaho the Kindle let’s me read a little longer before turning on the house lights. This probably saves me 1 or 2 cents a year.

Physical Library Size

The Kindle does have the advantage of being able to store my entire Kindle library wherever I am… not only is it smaller than 99% of my books, it can store all of my books in that space.

Fonts

90% of paper book publishers choose great fonts–but some don’t. For some reason some publishers think their book needs a sans-serif font, or they pick a huge font, or too small a font, or the kerning is not normal. It bugs me! If you get the Kindle version you can override the publishers horrible font decision. As an added bonus the font-size is adjustable so I can read anything without glasses.

Quality

I always prefer a good hardbound paper book to an eBook, however I’ve noticed lately a lot of authors are using cheap (self-publishing?) services–it seems to me the books are printed on demand and the quality is sometimes bad–I’ve had books that–the best way I can describe it is the book feels like I’m holding some ad-hoc document put together at a business conference rather than a book. I’ll often opt for an eBook if I see the author is using a self-publishing service (not all self-publishing books come this way–I think it’s just a quality control issue so it’s a hit and miss).

Enjoying Books with Others

The social aspect of eBooks is poor. Often when I’m on an airplane or a friend is at my house they’ll show interest in a book I’m reading or I have on the shelf and it makes a great conversation starter. You just don’t get that with Kindle books because nobody can see what you’re reading. Kids love physical books and will spend hours poring over maps, illustrations, and pictures which would be boring on a tablet. I can easily give a paper book to a friend. While Amazon has some provision for lending it’s very limited and it’s not as simple as handing your friend a book.

Highlighting and Taking Notes

For highlighting it’s a wash–the Kindle is sometimes a bit finicky when I try to highlight a passage and sometimes gets the wrong portion highlighted but for the most part I can get it. I always read a book with a pen or pencil but I find underlining a passage without the line going through the words to take a little more effort. For taking notes in the margin nothing can beat pencil or pen on paper.

Diagrams and Illustrations

Diagrams are pictures are generally bad on eBooks. For simple graphics it does fine. But if the book has illustrations they don’t look as great because the screen is smaller and you lose color.

Also, the Kindle completely fails at tables… this table below has data that is illegible on the Kindle… it’s too small to read and there’s no way to rotate it into landscape mode.

Flipping Through Pages

The Kindle is useless here. Even in the flip through the pages mode the e-ink display takes too long to refresh. A real book is much easier–plus I remember the layout of a page and generally know what I was looking for was in the 1st quarter of the book so can find it in seconds.

Searching

Here the Kindle shines. If you are looking for a keyword or phrase you can find it very quickly.

Visual Indicators of Progress

This is a big deal. I am very spacial and use the physical feel of how many pages I have read and how far to go as part of my memory. This is all lost on eBooks. With paper books it’s easy to see your overall progress at a glance, and if you want to thumb a few pages ahead to see when the chapter ends it takes half a second. With an eBook I get something like location 675 or 24%. That’s meaningless to me. A progress bar might be nice! Something visual and not just numbers. Even web-browsers have scrollbars!

Reading Books as a Group

When reading books for study with others eBooks fail–I tried this once but everyone else was referring to page numbers and I couldn’t get page numbers out of my kindle.

Free eBooks

Amazon has a lot of free Kindle books for Prime members. I’ve found the free books aren’t really that good so not much of a gain.

Free Classic Books

There are a number of great classic books you can download from the Guttenberg project, this may save you from purchasing a few paper books.

Updates to Books

Some of my more technical books have received free Kindle updates when the author chooses to update the text. This is a benefit in my mind. I think it would be better if the Kindle would highlight the differences.

X-Ray

One nice feature on the Kindle Paperwhite is the X-Ray. You can enable it for the page you’re on and it will tell you about the characters and give you some context (if you’ve forgotten the previous chapters or missed it).

Newspapers

You can read newspapers on the Kindle. But it’s worthless. The Wall Street Journal digital subscription is completely separate from the Wall Street Journal Kindle Digital Subscription. I’m not going to buy a Digital subscription for both my computer and my Kindle.

Synchronization

One great thing about eBooks is I can read them on my Kindle, then bring up the book on my computer to review my highlights while typing up notes–but it’s a hit and miss. This works for Amazon books I bought from the Amazon store. But if you buy Kindle formatted books from not Amazon there’s no way to get them to open up in the Kindle for PC program (even though they are available in Kindle for Android). Very annoying.

So, What’s Better? Kindle eBooks or Old Fashioned Physical Books?

It really depends. I like both for different reasons. I do have a preference for Print Books and mostly because I can visually track progress and visually see the layout of pages and flip through them. Generally if it’s a book I’ll probably read once I’ll just get what is cheaper… but obviously some I’m going to insist on getting the physical version. One feature that Amazon does for /some/ books is if you buy a physical, you can get the Kindle version for free, or heavily discounted. I do hope that this becomes standard practice going forward–that’s the best of both worlds.

Of making many books there is no end, and much study is a weariness of the flesh. The end of the matter; all has been heard. Fear God and keep his commandments, for this is the whole duty of man. For God will bring every deed into judgment, with every secret thing, whether good or evil.

Why You Should Have a Homelab

In 1998 my friend gave me a RedHat Linux CD. I spent hours each day experimenting with Linux–I loved it. 2 years later I’m in a room with 30 other students at a University applying for the same computer lab assistant job–I’m thinking my chances are grim. Part way through the mass interview a man walks to the front of the room and asks if anyone has ever used Linux. I raise my hand–I’m the only one. He takes me out of the interview for the lab assistant job, introduces me to the department director. They took me out to lunch. By the end of the day I had my first job as a Systems Administrator.

Learn things on your own and it will broaden your opportunities.

One of the best ways to learn about systems, applications, and technology is starting a homelab. A Homelab can give you an enjoyable, low stress, practical way to learn technology. A homelab will also help you find out the technical areas in which you are interested. It’s also practical in that you can use it to service your own home.

Here’s 7 Ideas for Your Homelab

1. Router / Firewall

The most essential piece of equipment will be your router. I started out with consumer routers that I’d flash to DD-WRT / Tomato but now I use a virtual pfSense router. Routers are great to learn about DHCP, DNS, VPN, Firewalls, etc. I discourage using the router provided by your ISP, they’re usually not very capable and often not secure. In most cases you can buy a DSL or Cable modem instead of the ISP provided modem combined with the router. One inexpensive physical router I’d recommend is the Ubiquiti EdgeRouter X. Ubiquiti provides free software updates (their model is you buy the hardware and the software is free), and you’ll get a handful of advanced features–it’s a very capable router and much better than a typical consumer router–to step up from Ubiquiti you’d be going to pfSense, Juniper or Cisco.

2. Storage

The main reason I started my homelab was storage. I was taking a lot of family pictures and videos and wanted to save them. I know there are cloud services, but at the time they were expensive, and then you’re sort of trusting that provider to not delete all your photos or get bought out by a larger company and shutdown.

Then I started using VMware. I needed faster storage with more IOPS. One of the best Homelab storage solutions is ZFS. ZFS takes the best of filesystems, and the best of RAID, and combines them into a software defined storage solution that I’ve not seen any hardware technology able to match. Two popular free ZFS appliances I like are Napp-It (based on OmniOS) and FreeNAS. OmniOS is a fork of OpenSolaris and is very robust and has tight integration with ZFS.

I’m currently using FreeNAS which is the free open source version of iX System’s TrueNAS which is used by organizations of all sizes–from small businesses with a few TB of storage to large government agencies with PBs of storage. FreeNAS has done a great job at technology convergence. It is both a NAS and a SAN allowing you to try both approaches to storage (I prefer NAS because it takes better advantage of ZFS, but many prefer using SAN and there are benefits and drawbacks to both), it also has many built-in storage protocols: FTP, iSCSI, NFS, Rsync server, S3 emulator, SMB (Windows file server), TFTP, WebDav, it can join AD, it can even be an AD DC (if you like living on the edge) it has a built-in hypervisor (bhyve) to run VMs for whatever you want. This is now marketed as hyper-converged storage. All of it is completely free. You can build your own FreeNAS server like I did, or get started with a FreeNAS Mini from iX Systems.

A few years after I learned ZFS for home, my employer was looking for a new storage solution so having this knowledge and experience was helpful. I was able to determine one vendor with a traditional RAID solution didn’t handle the RAID-5 write-hole problem properly.

3. Virtualization

Virtualization allows you to run multiple virtual servers on the same piece of hardware. VMware is king in the small to mid-size business hypervisor market, and VMware offers their hypervisor for free. The free version is just like the paid versions except you won’t be able to use some features (most involving high availability and fail-over with multiple servers). But you can learn most of the concepts and features of VMware. I’ve tried to use a number of hypervisors but I always come back to VMware. I consider VMware my basic infrastructure. From there you can learn about other things like networking, storage, and play with any OS or Linux distribution you want to.

Knowing VMware was hugely beneficial, I’ve implemented it for several businesses, and one of my previous employers. And knowing how it works means I can discuss the VMware stack intelligently with the ops team.

4. Networking

A Homelab without decent networking won’t get you far. Fortunately if you use VMware you can leverage it to use virtual network switches. For physical switches I really like the Unifi products. They are simple enough for non-network engineers like me. Everything can be configured using the GUI. Unifi exposes you to managed switches, central management (with the Unifi controller), VLANs, and PoE (Power over Ethernet), port trunking, port mirroring, redundant paths with spanning tree, etc.

I started with this little 8-port switch (4 are PoE ports). I also added a 24-port switch so I could learn how to do setup a LAG and configuring VLANs across multiple switches (which was really simple using the Unifi interface). I also like Unifi’s philosophy–they sell you the hardware but the software is free–which means you don’t pay for maintenance or support but continue to get free updates. In a homelab you may not need to go crazy on VLANs, but separating your main network from your IoT devices may be prudent.

Learning how to setup VLAN tagging, and link aggregation and understanding how networking works helps me communicate better with the network engineers when discussing design and deployment options–they may be working on Juniper or Cisco equipment but I know the concepts of what they’re doing.

5. Wireless APs

Having a robust wireless setup is also a necessity for a homelab. If you have a large house you get to setup multiple APs and make sure they can handoff connections. I use a Unifi AP Pro (I just use one because that’s all I need to cover my house, but if you can find an excuse to have 2 or more I’d recommend it since you can practice rolling updates without downtime, wireless handoff, etc.). These are managed by the same Unifi controller as the switches. I first gave them a try because I read Linus Trovalds uses Unifi APs, and they seem to be highly rated by tech professionals–and now I don’t think I’d go back to anything else.

6. Network Monitoring

It is hard to maintain a reliable network and application stack without monitoring for failures. There are hundreds of network monitoring solutions and it really depends on your needs. The most widely deployed solution is Nagios. I have had that on my Homelab, but lately I’ve been using Icinga because it’s simple and it integrates into Ansible.

7. Infrastructure Automation

Automating your infrastructure may not make as much sense in a small Homelab, but it does make sense to automate any task you do repetitively or a manual task that could be automated. For me, this was installing updates, deploying servers and renewing SSL certificates with Let’s Encrypt. To manage this I use Ansible which is one of the most well thought out infrastructure automation tools I’ve seen. Ansible can manage Linux and Windows servers. Learning infrastructure automation, especially if you do it using version control and CI/CD tools like Azure DevOps (you can get a free account for up to 5 users with unlimited private repositories) is a great thing to learn for your career if you’re interested in the DevOps world. The book, Ansible for DevOps by Jeff Geerling helped me get started. I suggest getting the eBook since he has been known to provide updates to the book (not sure if he will continue to provide updates, but just in case).

At work we completely automated the deployment of Linux servers using Ansible–infrastructure as code. It took a month of investment but it paid off big time with developers now being able to deploy VMware VMs at will with Ansible by making a Git Pull Request, our entire fleet of servers is updated automatically, and our server and configurations are all consistent. This replaced an old process of waiting several weeks for a VM to be provisioned and configured by hand.

Bonus homelab application server ideas…

Minecraft Server — popular Java game–it’s like playing with Legos and a great way to get your friends together for some casual games.

Mumble Server – one of the best voice protocols for in-game communication.

Emby Media Server — Anyone that has kids realizes those flimsy blu-ray drives aren’t going to last long. It’s great to store and host movies, home videos, pictures, and audio.

Asterix PBX Server – VoIP Phone server (use Twilio or Flowroute for SIP trunking). Polycom makes great VoIP phones. With Twilio SIP Trunking you can have a real landline phone number with E911 capability for a few dollars a month–and if you get multiple phones you can use it as an intercom system.

Web Server (maybe start a blog) — I hosted this blog from a server in my house for years–until my ISP couldn’t handle the bandwidth. Now days you can also use a service like CloudFlare to act as a CDN which really reduces your bandwidth usage. Hosting your own blog is a great learning experience and gives you a place to log your homelab experiments, and share solutions to problems.

Backup server — I use a CrashPlan Business subscription to backup my FreeNAS server to the cloud (one of the main reasons I use a NAS as this would be less efficient with a SAN). BackBlaze B2 is another great option to backup FreeNAS.

…

There are many more areas than I listed, but I think the above is a good baseline to get started. Pick one area at a time–my homelab was built over many years–often the case is I will improve an area after a piece of equipment fails or I need to replace it for some other reason–that’s a great time to do research. If you aren’t sure where to start, pick the area that you enjoy the most. For areas you have no interest the best thing to do is something else–you’re probably not going to be great at something you don’t enjoy. Certainly a homelab isn’t going to be a substitute for real-work experience. But it does provide an environment to learn, experiment and enhance your abilities–and the great thing is since it’s your own lab you can learn things that interest you.

I think that’s the largest benefit of a homelab. To me it’s a playground. It’s a place put the love of learning into practice. It’s a place of freedom. Nobody else is dictating what you do here. It’s a place to have fun while enhancing your skill.

Do you see a man skillful in his work?He will stand before kings;he will not stand before obscure men. – Proverbs 22:29 ESV

Happy 501st Reformation Day! (and Happy Halloween)

In 1521 Luther was summoned to the city of Worms to be tried for heresy. Expecting an opportunity to debate Luther willingly went. But instead found no discussion would take place.

He was asked to answer two questions:

First question: With his writings on a table in front of him he was asked if the books were his. He answered in a barely audible voice, “The books are all mine, and I have written more.”

Second question: “Do you defend them all, or do you care to reject a part?”

Luther asked for time to think about it.

He was given one night.

The next day, Luther returned and was asked again, “Do you recant?” Luther tried to explain his views but was quickly shutdown. The official asked him to answer the question simply. Luther answered:

Since then your serene majesty and your lordships seek a simple answer, I will give it in this manner, not embellished: Unless I am convinced by the testimony of the Scriptures or by clear reason, for I do not trust either in the Pope or in councils alone, since it is well known that they have often erred and contradict themselves, I am bound to the Scriptures I have quoted and my conscience is captive to the Word of God. I cannot and I will not retract anything, since it is neither safe nor right to go against conscience. I cannot do otherwise, here I stand. May God help me, Amen.

Luther threw up his arms making the gesture of a victorious knight, slipped out and went into hiding. Obviously, Luther was pronounced a heretic. The Reformation had started.

The central issue of the Reformation is this question. How does a person get right with God?

The answer to this question can be summarized into the 5 principles below:

Sola Scriptura – Scripture Alone

Scripture claims to be inspired by God. It is without error, authoritative, and sufficient.

All Scripture is breathed out by God and profitable for teaching, for reproof, for correction, and for training in righteousness, that the man of God may be complete, equipped for every good work. – 2 Timothy 3:16-17 ESV

There is the ever changing doctrines, beliefs, and ideas of man. We have various political views, ideologies, philosophies, worldviews, and thoughts that change drastically every few hundred years. Then there is Scripture. It stands unchanging. It has been studied, scrutinized, and examined more carefully than any other publication in history–for 1600 years. It is God’s Word. Many claim the Bible is full or errors. Many claim it has been translated multiple times and the contents changed like the telephone game. However, the method of Scripture transmission through the ages has a consistency check to detect and correct changes.

Imagine, you create a file. Then you make 10 copies of that file onto floppy disks, CDs, or USB drives, and give it to your friends, they each make 10 copies and give it to their friends, and it gets distributed throughout the world. 1000 years into the future you will find hundreds, maybe thousands of copies of the data. They might be imperfect–you may have partial discs, bitrot, on some of the copies someone may have lost power in the middle of a copy and it corrupted or skipped data, some people made malicious changes, not all the copies agree with each other. But even without the original you can know the contents of the original file. As long as you have enough copies and a history of those copies you can reconstruct the original. Even if mistakes are made in all of the first 10 copies of the original (as long as the same mistake is not made in all 10) you can reproduce the original–like a RAIDZ array can reconstruct damaged or missing data by using the parity data from the other drives.

Scripture was transmitted through the ages in such a manner that it could not be changed from the original undetected. Even though some copies of scripture were damaged, missing parts, had a copy mistake, etc. Most discrepancies are minor and don’t change the meaning of the text or change any doctrine. With vast geographic distances and no modern means of communication there would be no way to collude a malicious change. When we do see a difference between various families of manuscripts, we know what changed. That means we know what the words of God actually are.

I’ve read on random internet forums that the Bible is a translation of a translation of a translation and so much has been lost in all those translations. That’s not true at all. Most English Bibles are a direct translation into English made by a team of scholars who have access to the available copies of the manuscripts in the original languages from various regions. The Bible is translated directly from the original languages written in Hebrew, Greek, and Aramaic into English so you can read it. The English Bible you have today, if made by a trustworthy translator, is reliable. God promises His word will stand forever:

Sola Gratia – Grace Alone

Man cannot save himself. God saves sinners. We are absolutely incapable of pleasing God. We have all sinned against a Holy God. When God created man, He created us in His image–we are held up to God’s standard of righteousness and because of sin we all fall short. Ask yourself these questions: have you ever said God’s name in vain, have you ever hated someone, have you ever stolen, have you lied, have you ever looked at a woman (or man) with lust? Answer those questions honestly. Have you done any of those? “Yeah, but everyone else does it…” doesn’t matter about everyone else. What about you? If you stood before God today, and He was judging you by the Ten Commandments, would He say you are Guilty or Not Guilty? I don’t get a pass on this either, I am as guilty of breaking God’s law as anyone.

There is nothing you can do to get a not guilty verdict. You may decide that from this day forward you’re going to do good, you’ll stop sinning, love others, give away your money, feed the poor, go to church, etc. It doesn’t matter. God considers the good things you do like a polluted garment (Isaiah 64:6). Your “goodness” is worthless to God. You will still go to Hell.

As it is written:None is righteous, no, not one;no one understands;no one seeks for God. — Romans 3:10 ESV

One of the most important distinctions between false religions and Christianity is this: false religions teach that you can earn merit. Essentially they teach you must do good things to get points from God to earn your salvation–as if life is some sort of video game. Doing good to get to heaven is probably one of the largest misconceptions about Christianity. The Bible teaches the exact opposite: You cannot do anything to save yourself from Hell. But, there is God’s Grace. God’s unmerited favor towards mankind:

For by grace you have been saved through faith. And this is not your own doing; it is the gift of God, not a result of works, so that no one may boast. — Ephesians 2:8-9 ESV

Sola Fide – Faith Alone

By faith, and faith alone are we justified (made holy) before God. When Christ died on the cross, He paid the penalty for our sins.

Justification is about our legal position towards God. The means by which we are justified is through Jesus’s death–when God looks at us, He sees the perfect life of Jesus. He sees that Christ has paid the price for our sins–Jesus’s righteousness is imputed to our account–which means through no power of our own we are declared righteous before God–despite being imperfect. This is what our faith is in–and by faith we do not mean a blind faith, but confident belief in what God has promised in Scripture, and God’s ability and integrity to keep those promises. That is what we must have faith in. And faith alone in Jesus Christ is the means by which we are justified so that we can have eternal life.

Therefore, since we have been justified by faith, we have peace with God through our Lord Jesus Christ. — Romans 5:1 ESV

Solus Christus – Christ Alone

Jesus is the God-Man. There is none like Him. He is eternal, existing outside time and creation with God the Father, He is the Creator of the Universe, He is God Himself, the second person of the Trinity. ~2000 years ago He stepped into His own Creation and became a man and now possesses two natures: God and Man. Jesus lived the perfect life–the righteous life we could not live as a man. And took the punishment we deserve for our sins: death. Christ is the only means of salvation. Christ’s humility cannot be overstated–we are talking about God the Son, who has always pre-existed with God the Father in the presence of God’s glory coming to earth to be humiliated and die–if Jesus had come to earth as King and ruled the world, lived in the best place and had the finest the world had to offer with servants at His side to take care of any need or task, it would be an insult compared to the glory He deserves–yet he came to die.

A common misconception is Jesus’s death was some work of Satan and something outside of God’s control. No. God sent Jesus to earth specifically to die for our sins to offer us salvation from our sins.

Jesus died on the cross, was buried, and then rose again. There is no other religion, no other method to obtain salvation, other than Jesus to become right with God:

And there is salvation in no one else, for there is no other name under heaven given among men by which we must be saved. — Acts 4:12 ESV

Soli Deo Gloria – To The Glory of God Alone

The salvation of those who love God is an amazing, beautiful work of God. It is not something we can claim even partial credit for.

I have heard the analogy of salvation being like God throwing a life saver out into the ocean to one screaming for help and all one has to do is reach out and grab it to be saved. That is exactly what the work of salvation is not like.

We are not drowning begging for help. Rather, we are all enemies of God, storming His castle, with no inclination of good in us, and God chooses to save us through the Gospel–which is the good news that Jesus Christ died on the cross to take the penalty for your sins. By believing, by having faith in Him, you will have eternal life. There is nothing for us to do in regards to our salvation, because the glory is God’s, and God’s alone.

I am the Lord; that is my name;my glory I give to no other. — Isaiah 42:8a ESV

For from him and through him and to him are all things.To him be glory forever. Amen. — Romans 11:36 ESV

What is DNS Content Filtering?

A DNS Based Content Filtering service can prevent certain websites from loading on your network. Most services can filter by specific categories like malware, phishing, pornography, etc. Unlike some content filtering which can introduce security risks, DNS filtering does not intercept traffic between you and the website you’re visiting. It doesn’t require installing any software on your computer or device making it one of the safest ways to filter web content.

If you you accidentally typo a popular domain (such as typing .cm instead of .com) it would normally take you to a phishing site. A DNS filtering service would block your computer by returning an NXDOMAIN (domain does not exist) instead of the IP address effectively blocking the website from loading. The same technique can be used to prevent any undesirable category such as malware, pornography, adware, etc. from loading on your network.

The other benefit of using a DNS filtering service is it can force certain search and media services (like Google and YouTube) into safe mode preventing anyone using your network from even seeing adult content in their search results.

Why Should I use One?

It’s not only a wise way to protect yourself from malware and temptation, but also when letting guests on your WiFi network–you don’t have to worry (as much) about what they’re doing, and also a good idea when you start letting kids online. DNS filtering doesn’t take the place of parenting, and anyone with a little technical skill can bypass it, but it may help prevent your family and anyone using your network from accidentally stumbling across bad sites. If it prevents one cryptolocker infection it’s worth it.

I think families, churches, home networks, small businesses, organizations, schools, large enterprises, and governments could benefit from DNS filtering. You may not want to go overboard blocking content about illegal drugs and gambling, but at the very least you probably don’t want malware on your network!

Two DNS Filtering Services

I use two DNS content filtering providers services: OpenDNS and CleanBrowsing. Both have simple instructions to get started so I won’t repeat that here. Both are free, work well, and my decision to use one or the other on a particular network just depends on the situation–although in most cases either would be fine. It’s nice to have multiple options.

OpenDNS

OpenDNS has been around since 2006 and was acquired by Cisco in 2014. It offers several free plans and some paid options as well:

OpenDNS Family Shield(Free). Very simple–just set your router’s DNS servers to 208.67.222.123 and 208.67.220.123 and it’s pre-configured to block malicious and adult content.

OpenDNS Home (Free). For more advanced control, allows for granular category filtering as seen in the screenshots below. If your ISP has a dynamic IP you will need to use a DDNS client to update OpenDNS with your public IP. Below are some screenshots to show the granularity:

OpenDNS Home VIP($20/year) — Very affordable and adds the ability to white-list specific domains if they’re on the block list.

Cisco Umbrella — For businesses and larger enterprises.

CleanBrowsing

CleanBrowsing is a fairly new service, starting in February of 2017.

It offers three easy free filtering plans and 2 paid plains:

Security Filter (Free) – Set your router’s DNS to 185.228.168.9 and 185.228.169.9 to only block malicious domains (phishing and malware).

Adult Filter (Free)– Set DNS to 185.228.168.10 and 185.228.169.11 to block Adult domains, set search engines to safe mode (also includes the security filter).

Family Filter (Free)– Set DNS to 185.228.168.168 and 185.228.169.168 to block access to VPN domains that could be used to bypass filters, mixed content sites (like Reddit), and sets YouTube to safe mode (includes Adult and Security filters as well).

Much better Test Results Blocking Phishing Sites: CleanBrowsing blocked 100% of phishing sites on 3 out of 4 tests beating out OpenDNS in every area. On the real-time test it allowed 1 out of 12 sites through, however OpenDNS only blocked 2 out of 12 sites.

Both OpenDNS and CleanBrowsing have very fast DNS resolution rates (probably faster than your ISP), with CleanBrowsing resolving slightly faster for me but within milliseconds of each other. I think either service is worth using.

I have made a covenant with my eyes.How then could I look at a young woman? — Job 31:1 CSB

Kris and I finished watching the Genesis and the State of the Culture speech that Ken Ham and Dr. Georgia Purdom gave at the University of Central Oklahoma.

To give some context, the LGBT group at UCO tried to stop Ken Ham from giving this speech. At some point the student body that invited him was pressured into revoking the invitation [1]. Later Ham was re-invited by UCO President Don Betz.

This is a well done overview of what Christians believe and why it is reasonable. Ken ham gets to the heart of the issue which is foundational and shows how making your foundation God’s Word or Naturalism affects ones beliefs on race, gender, abortion, etc. After Ham’s presentation, Dr. Purdom makes the distinction between Natural Selection and Evolution (this is an important difference that even my University professors were not all aware of) and gives several examples of how Evolutionists do not have a mechanism to generate novel traits which is necessary for molecules to man Evolution.

I highly recommend watching it:

“So, one of the things I am saying to you is from a biblical perspective we’ve got the answer to racism. We’re all descended from one man. We’re all equal before God. We’re all sinners. So we should be treating each other as people who are all family.”

— Ken Ham

Evolution’s Tree

Creation’s Orchard

“There are evolutionists that still say junk DNA is junk even though there have been multiple observable science research studies showing it is not, showing it has function. It shows you that this is a worldview issue. Even though the evidence is staring them in the face, they refuse to believe.”

I recently switched to MobaXterm Professional from PuTTY. And I’m not looking back…

A PuTTY Alternative

I had just re-installed Windows 10 to fix an updating issue. As I was downloading PuTTY I thought: there has got to be something better than PuTTY. PuTTY is a good program, but it doesn’t do four things for me:

Automatically save the SSH session

List of recent servers I’ve SSHed into for a quick reconnect. I know this is nitpicky on my part, but I don’t really remember all my server hostnames or IP addresses.

SFTP. I just want to drag and drop files between the terminal and file explorer without having to open another program!

If I make changes to a saved session in-flight and I don’t remember to save it (such as setting a keepalive) PuTTY forgets it.

I looked at and tried quite a few options. KiTTY, MobaXterm, mRemoteNG, RoyalTS, SuperPuTTY, XShell6, Bitvise, SmarTTY, Solar-PuTTY, and SecureCRT. I ended up buying MobaXterm.

What I Like About MobaXterm – A Quick Review

Start Screen

The start screen is simple and useful… open MobaXterm and start typing a hostname… if you’ve connected to that server before it will auto-complete, if not it creates a new session.

Along the left is a list of servers which can be organized into folders and the icons can be customized. Main screen shows the last 9 sessions for quick access.

New Sessions

MobaXterm supports a number of protocols:

SSH

Telnet

Rsh

Xdmcp

RDP (yes, it can even manage Windows RDP sessions)

VNC

FTP

SFTP

Serial

File

Local Shell (which includes Ubuntu Bash WSL if you have it installed, Powershell, Bash on Windows, normal DOS Prompt)

Browser (opens a browser)

Mosh

S3

Integrated SFTP File Transfers on the Terminal

SSH into a server and the left pane shows an SFTP session which automatically follows where I am in the terminal and allows dragging and dropping files back and forth between file explorer! No more having to open up WinSCP just to transfer a quick file.

Files can also be opened directly and edited using a built-in or an external editor.

X11 Forwarding

X11 forwarding works out of the box with no setup. Below all I did was open an SSH session to my Linux VM running CrashPlan, ran “CrashPlanDesktop” (which is a graphical program) and it opened up the window locally in Windows.

One of my favorite programs in the world, Minesweeper, no longer comes with Windows 10. It’s such a classic I don’t know what Microsoft was thinking by removing that. But… no problem. I can now run Gnome Mines on Windows via X11 Forwarding!

Terminal

The terminal itself is actually PuTTY under the hood but with some added features. There’s a place to configure key words that if they show up on the terminal are highlighted in certain colors; the defaults are useful when reviewing logs. Terminals can be tabbed, or split horizontal, vertical, or a grid of 4. You can also open multiple MobaXterm Windows. Terminals can also be dragged off to float (more like PuTTY terminals do). Right-click can be configured to paste like PuTTY or provide a menu (also if pasting multiple-lines it will display a warning which is nice.). If you don’t like the Windows 10 everything is flat look or you want a Dark Theme or want it to look like you’re on OSX there are a plenty of skins to chose from…

Setting up SSH tunnel port forwarding is easy…

For storing passwords and SSH key authentication MobaXterm can manage that and also save passwords (if you’re using something that uses password authentication which you shouldn’t be) securely. I use an external ssh agent and it handled that well.

Extra Utilities

And MobaXterm comes with quite a few handy programs and utilities… a variety of servers which is useful if you need to temporarily setup a quick Iperf or TFTP server. Also included are Macros, and a variety of misc tools such as a Network Scanner, Port Scanner, etc.

A fantastic feature is the ability to run local terminals. I can run a DOS Prompt, PowerShell, and Ubuntu Bash (WSL) terminal inside MobaXterm.

What Could Be Better

A few features that are missing:

The SFTP pane should elevate to root when I “sudo su”Update: MobaXterm told me to use the SCP protocol instead of SFTP and there’s a quick button in the SFTP pane to sudo su. This works.

I’d love to be able to open up a VMware ESXi VM console from MobaXterm.

Would like to have an option to use integrated SFTP with Mosh

The cost structure is very reasonable at $69 for a perpetual lifetime license but after the first year support/maintenance is 80% of the cost of the license. I think the price is more than worth it but I’d love to see a lower maintenance price for home users or businesses under a certain size.

Some SSH settings can’t be defaulted and have to be explicitly set on each session. I prefer to never lock the terminal title, and also I always want the SFTP directory to follow the directory in the terminal but neither of those can be set globally. Fortunately the session remembers the settings so you only have to set it once per host, but there should be a global default.

RDP settings should have configurable global defaults… I never want to share my local drives or printers during an RDP session so have to uncheck those when first setting up a session.

It is very similar in function to lever action rifles (lever action works the same, cross-bolt safety is the same) so it’s a great starter gun to teach proper firearm handling and safety. It was $18 at Walmart but has quality where it counts. The stock is real wood with an engraved image of a buck. The barrel and receiver are metal. The only plastic part is the lever but it’s rugged and shouldn’t break. If properly cared for I don’t see why it wouldn’t last several generations.

Daisy Model 105 Buck

There’s no recoil and it’s quiet enough to shoot in the backyard. The lever action does take some effort to work. Eli struggled with it at first but after a couple tries he doesn’t have any trouble cocking it himself. The gun shoots 0.177 caliber BBs which run a couple of dollars for 2,400 rounds.

I found out Daisy has been around since 1886 with an interesting history. They’ve been making air guns for a long time. They started out as a windmill company that started offering a free air rifle to farmers with the purchase of a windmill. A few years thereafter they stopped making windmills.

Daisy advertises it for age 10 and up, I think the age is listed for legal reasons more than size. For fit I’d say it’s sized for 6 to 10 year olds at 30 inches long. For younger kids new to guns the shorter length will be easier to manage. If I was buying for someone over age 10 I’d probably opt for the infamous Daisy 1938 Red Ryder which has a longer stock and is 35 inches long.

BB’s can ricochet so keeping a safe distance from the target as well as safety glasses are a must. The instruction manual has directions to build a target stuffed with newspapers with magazines at the back that will absorb BBs. I placed the target in front of my shed but in case he misses the target I set behind the target a sheet of plywood which I angled down to cause any ricochets to bounce down into the ground. So far this has worked well. For targets we use milk jugs or print our own.

I think it’s good to teach children how to use guns when they’re mature enough. It gives them a chance to develop responsibility and character while having fun and learning a useful and important skill.

The NRA has a great Gun Safety Rules page. Before Eli could have ammo I taught him 3 safety rules:

Always point the gun in a safe direction especially away from people, even when it’s unloaded or the safety is on.

Always leave the safety on until you’re pointed at the target. I make him turn the safety on before he puts it down or hands it over to me.

Always keep your finger away from the trigger until you’re ready to shoot.

I’ll teach him more as he gains skill. We treat the BB gun the same as my firearms. The gun is unloaded after each use and is stored in a safe location where nobody else can get to it, separate from the ammo. I think it’s good to teach proper gun safety and responsibility from the start. The best way to do that is by example.

Like this:

Happy World Backup Day! Here’s a quick little Ansible Role I wrote to automate backup configuration for hordes of servers using Rdiff-Backup from an Ansible inventory file. If you have no idea what I just said you may want to skip to “I’m Confused” at the very bottom of this post.

What does the Rdiff-Backup Ansible Role do?

Creates a folder on the backup storage server to store backups.

Creates a backup script on the backup server. This script will use rdiff-backup over ssh to backup every server on the list (below) and prune backups older than 1-year (default).

Adds/removes servers in an Ansible inventory file to a backup list which the backup script calls as servers are provisioned/decommissioned (the script will not delete backups on a decommission, only stop taking them).

Installs the rdiff-backup program on both the client and backup server.

Generates an SSH key-pair on the backup server and adds that public key to the authorized key file on each client to allow the backup server to ssh into the clients.

Scan ssh-key from client and add it to known hosts on backup server

Create a cron job on the backup server to run the backup script once a day.

Once the playbook has run all servers will be configured for backup which will occur at the next cron job run (defaults to 01:43 am).

The above playbook should be added to your site config so it is run automatically with the rest of your Ansible playbooks. It would also be wise to have something like Nagios or Logcheck watch the logs and alert on failures or stale log last modified dates.

The backup script does not try to create an LVM snapshot and then backup the snapshot. That would certainly be cleaner and I may add that ability later. The default settings exclude quite a few files from the backup so make sure those exclusions are what you want. One thing I excluded by default is a lot of LXC files. If you’re using LXC you may want them. Also always test a restore before relying on it.

Obviously, test it in a test environment and make sure you understand what it does before trying it on anything important.

Check your backup strategy

This is a good day to check your backup strategy. A few things to consider:

System backups are important, not just the data files. You never know what you’re missing in your Document only backups and restoring service from system backups is much faster than rebuilding systems.

Frequency. If you can’t afford to lose an hour of work backup at least every hour.

Geographic redundancy. Local fires, hurricanes, fires earthquakes can wipe out multiple locations in cities all at once. Keep at least one backup in a separate part of the globe.

Versioned backups. On Monday you took a backup. On Tuesday your file got corrupted. On Wednesday you overwrote Monday’s backups with Wednesday’s backup. Enough said.

Test restoring from your backups (it’s good to test at least once a year on World Backup Day) to make sure they work.

Encrypt. Make sure your backups to cloud services, insecure locations are encrypted (but also make sure you have provisions to decrypt it when needed).

Cold storage. Keep at least one backup offline. When a bug in your backup program deletes all your live data and your backups you’ll be glad you did.

Keep at least 3 copies of data you don’t want to lose. Your live version (obviously), one onsite backup that will allow you to restore quickly, and one offsite backup in a far away state or country.

I’m Confused

You might want to backup your computer. I’d suggest looking at CrashPlan, SpiderOak, or BackBlaze which are all reputable companies that offer automatic cloud backup services for your computer. The main thing you want to look at for pricing is how much data you have vs the number of computers you have. CrashPlan and BackBlaze charge by the computer but offer unlimited data so they would be ideal if you have a lot of data but few computers. SpiderOak lets you have unlimited computers but charges you by how much space you use making it ideal if you have little data and many devices.

. <-- this is a dot

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 158 other subscribers

Email Address

b3n.org is a participant in the Amazon Services LLC Associates Program, an affiliate advertising program designed to provide a means for sites to earn advertising fees by advertising and linking to amazon.com