Monthly Archives: April 2016

In our previous post, we were awaiting the completion of burn-in testing for the FreeNAS solution that we had constructed. Approximately 48 hours was required to complete the recommended short SMART test, the badblocks test, and the long SMART test on all four drives. With the firmware fully updated for the motherboard, out of band management controller, and the disks, we were off to the races. Initial tests and performance numbers were very impressive, but we encountered some unexpected issues and challenges within the first week of operation.

1.) A single early life failure on one of the four Seagate Constellation ES.3 drives. The beauty of taking a few minutes to set up e-mail alerts resulted in immediate notification when the drive failed and the pool entered a degraded state. As we were within the 30 day return period, we RMA’ed the drive and awaited receipt of the replacement unit. The new drive was definitely from a different batch and had the most recent firmware. Validation was successful and the fourth drive was added as a replacement for the first pool.

2.) There’s a CrashPlan plug-in available for FreeNAS. If you take that statement at face value, it wouldn’t appear any different than comparable, community-developed offerings for competing, commercial off-the-shelf solutions. However, the underlying story is far more complex and involves sifting through multiple processes to achieve a fully updated and operational jail with the current version of CrashPlan. Many of the posts we found were helpful and fairly accurate, but some commands and file names have changed over time. We have been performing off-site backups now for close to a week without issue, and reboots of the NAS to apply updates have not impeded operation of the backup solution in any way. The jail is rock solid and the services do not encounter issues when starting.

Having had enough time with the solution under our belt to formulate a solid opinion, the pros far outweigh the cons. The scheduling of standard maintenance tasks is very robust, development and updates come at a very fast pace, and the community – while terse if you didn’t take the time to RTFM – has enough experience to help resolve any issues that you may encounter. The notifications that can be e-mailed as soon as a non-standard event is detected go a long way in enabling the ability to proactively solve issues before they have a chance to take away the underlying data. The reporting mechanisms are very robust and easy to understand. Advanced networking functions have enabled multi-VLAN support and facilitated the appropriate path configuration for iSCSI storage without considerable effort, which is a significant benefit for our virtualized environment.

The fixes we’ve witnessed while researching the list of updates have promptly addressed errata for configurations that are even more extreme than ours. We’ve actually compiled yet another process that we’ll publish and share related to all things CrashPlan within a FreeNAS jail. The pending release of CrashPlan 4.7 in the near future will result in us taking a pause prior to publishing this body of work so that we can validate paths and file names for the newest packages. This will ensure the procedure will be current and valid for the foreseeable future. We’ve even uncovered some additional “gotchas” that occur when you take over an old CrashPlan backup set that had a different configuration (disk layout, IP address, etc.). Stay tuned!

The ability to consume copious amounts of media, when combined wanting to protect critical data, has developed a considerable market for network-attached storage devices and appliances. Many different tech sites have provided benchmarks and recommendations based upon the outcomes of various tests. The reality is that most of the leading consumer offerings do so much more than just “storage”. Synology, for example, provides app containers via Docker, and has introduced support for btrfs. The additional layer of protection against digital bit rot and potential corruption using legacy file systems can be mitigated with this file system. Other vendors excluding Netgear may not have a clear road map that defines when they’ll introduce support for next-generation file systems.

The only other primary alternative to commercial, off-the-shelf solutions pertains to building your own NAS. Depending on the feature set and level of expertise one has with available software and hardware platforms, it is within the realm of reality that a solution can be constructed with a superior performance profile and a feature list that meets or exceeds the boxed solutions. Prior experience with FreeNAS 8.x on a low cost platform that had limited disk expansion capability was excellent from the administration and functionality perspectives, but the hardware lacked the necessary compute. Furthermore, the single NIC offered in this previous test environment was known to have issues.

In following along with the latest updates and releases, this felt like the right time to dive back into FreeNAS using some cost-effective new components, along with some components that were freed up from servers we’ve retired last year. Before we committed to this endeavor and the construction of our solution, extensive reading, scouring, and researching through the FreeNAS community forums helped bring us up to speed with some of the nuances related to system design and optimizing ZFS performance. The sheer body of work put together by active participants was incredibly helpful and ensures that newcomers avoid potential mistakes that can cost them their data.

Ultimately, our build used the components listed below. Cost optimization can be achieved using search engines, coupon codes, eBay or forums where selling/trading is permitted.

Enclosure: SuperMicro CSE-825TQ-563LPB

This rack-mount enclosure includes almost everything you’ll need for installation if it’s paired with a SuperMicro motherboard. While this unit only comes with a single 560W power supply, it will be more than enough to address a solution which uses all eight hot-swap drive bays. In addition to ensuring disks are easily removable in the event of a drive failure, this unit also provides 2 non-removable traditional 3.5″ slots for drives. Overall space within this enclosure is ample when the SuperMicro cables are used to connect the required HBA to the back plane. Four cables of varying lengths are included in the box. The thin design ensures they route cleanly through and minimizes the amount of slack. Another matching set of cables runs approximately $30 USD. If you want things clean, don’t be cheap. Make this investment.

Motherboard: SuperMicro MBD-X10SL7-F-O

As we had previously noted, Haswell-based offerings provide an incredible bang for the buck. While newer Skylake offerings are supported for FreeNAS, we erred on the side of caution and went with a proven platform that has an essential bonus that will keep costs and cooling considerations under control. The majority of the configuration of this board is standard fare for a server-class solution: 4 DDR3 memory slots, Socket 1150, dual Intel i210 gigabit NICs, and a dedicated port for out of band management. The bonus here is the inclusion of an integrated LSI Logic 2308 HBA. This controller, when flashed with the appropriate firmware, provides for trouble-free performance within FreeNAS. Concerns raised about the operating temperature of a dedicated HBA in a non-server platform are effectively mitigated with our selected combination. The SAS controller’s heatsink resides under the adjustable air shroud that’s included with the enclosure we’ve selected. The exceptional airflow provided by the removable (and fairly quiet) 80mm fans addresses operational concerns. You can buy a SAS HBA for close to the price of this motherboard. For us – getting the motherboard with the HBA baked in was a no-brainer.

Memory: 32GB Crucial ECC DDR3-1600 DRAM

This was one of the items that we had available from a prior server. It’s on the compatibility list for the motherboard, all modules were already tested and certified as good, and reusing this memory resulted in reducing the amount of e-waste we’re generating.

CPU: Intel Xeon E3-1225 v3

Another one of the items that we had on-hand. If you’re erring on the side of caution or plan on doing “things” above and beyond storage that will benefit from hyperthreading, the E3-1230 v3 will be the target entry point for a Xeon E3 CPU. It’s also possible to go downstream with a Pentium or Core i3, both of which support ECC memory.

Install target: (2) SanDisk Cruzer Fit 16GB USB Flash Drives

Form factor was the biggest consideration for our selection of this specific model. There is 1 USB 3.0 header on the motherboard itself that can be used in conjunction with these incredibly low-profile flash drives. The ability to create a mirror during install ensures that the least expensive component for this solution isn’t the single point of failure.

The premium for these units versus more recent and NAS-focused models has been reduced over the past few weeks. Multiple sources are now selling these for ~$170 USD per drive. The 5 year warranty, 7200 RPM spindle speed, 128MB cache, and available performance metrics will facilitate meeting the target performance profile for the disk layout we’ll be using. Research does state that 5400 RPM NAS drives will still provide enough performance for most use cases while running cooler and consuming less power. Our luck with Western Digital Red drives from two different NAS units has been less then stellar. Shipping five out of eight back due to unexpected early life failures was an experience that we do not want to repeat this time.

A few notes from our experience in constructing this solution:

1.) SuperMicro’s Motherboard-to-Chassis relationship information on their website isn’t fully accurate nor all encompassing. If you look up the 825TQ-563LPB, you’ll note that the motherboard we used isn’t listed as being designed for the case. With the removal of an adjustable plate in the chassis and a quick relocation of some of the mounts, this board fit beautifully. Apparently, we’re not the only ones to see the potential for this combination.

2.) The SuperMicro SATA Cable Set (P/N: CBL-0180L-01) should be purchased with the enclosure. One kit is included with the chassis, and the second kit will resulted in saved time, connections being made with the optimum length, and risks of thick cables impeding the re-installation of cooling fans will be eliminated.

3.) Ensure the necessary firmware has been applied to all hard drives and/or SSDs that will be used in the solution. While we may take for granted that enterprise-class solutions are capable of applying disk firmware updates while remaining attached to a RAID controller or HBA, this benefit does not exist in the consumer space. For the solution we’ve designed, the hard drives came with the FN04 firmware, and the most recent version is FN06. Transplantation of the hard drives from the enclosure to an available test bench was require in conjunction with customizing and modifying the vendor-provided tools. While Seagate states that their bootable Linux USB utility includes the firmware, the reality is far from that truth. The process we used to successfully update all four drives can be found here.

We’re currently performing the long SMART test on the drives, and will provide an update once we’ve completed the burn-in for the solution.

We’re not even halfway through the week, and we have signs of life from three significant players in the compute space:

nVidia GTC 2016’s keynote kicked off today, and the Pascal-based P100 accelerator was announced. Understanding that nVidia can build their next-generation product stack from the top to the bottom, the implications for their GPU technology are pretty considerable. First and foremost, this will be one of the initial commercial products to adopt HBM2 memory. The improvements in available memory densities using HBM2 trump the limitations encountered for the original HBM (AMD’s Radeon Fury line of GPUs) while maintaining the bandwidth advantages. The “supercomputer in a box” offering (a.k.a the DGX-1) is a technical tour de force that offers an impressive display of incredible performance at density. While the $129,000 price tag may initially limit adoption to the target audiences identified in the keynote, the potential to moderate price with overall GPU capability opens up sales to new markets at a later point in time. If the high-end of nVidia’s consumer and professional GPU lineup leverage a Pascal-based solution that isn’t too far cut down from the top-of-the-line P100, then we’ll all be in for a treat later this year.

AMD executed with a bold strategy by pre-announcing their Bristol Ridge APU. Although this is not the next-generation Zen architecture that many would have hoped for, it does demonstrate signs of life for the underdog beyond the minor clock refreshes and new coolers for their stagnant product line. The improved manufacturing process for these solutions, when combined with current generation memory technology, provide another bump in performance that benefits the value-conscious consumer. As long as it’s reasonably priced, offerings such as the HP model mentioned in the link may strike the right balance that resonates with consumers.

Speaking of HP, someone in the organization must have finally gotten the memo about producing an overabundance of mediocre laptops in a contracting market. With consolidation efforts underway by Japanese laptop manufacturers (Toshiba, Vaio, Fujitsu) to improve profitability in this new reality, stocking store shelves with an overabundance of hunks of plastic containing 720p screens, low end processors, and lackluster input mechanisms is the wrong approach. Taking a page out of the Dell XPS and Apple playbooks, HP went big with the latest iteration of its Spectre line. While the keyboard doesn’t look like much of an evolution over the mediocre offering in its business-class laptop line, the fact that they’ve created the slimmest solution produced thus far without considerably compromising the compute within the solution is astonishing.

Although the price touches (and will exceed once you maximize the configuration) MacBook territory, the laptop is certainly aesthetically pleasing. As long as the lessons of the past related to thermals (I’m looking at you HP EliteBook 8470w with your gigantic chassis and multiple replacement cooling solutions) have been accounted for and the laptop doesn’t throttle when you blink, this solution should help demonstrate to HP that people really do want to get their money’s worth when buying a premium laptop. Quality over quantity is how you earn the business and keep the customer base.