Building a powerful, cheap and silent Linux NAS and HTPC server

Introduction

I am currently using an old IBM Thinkpad T42 as a server and NAS for my desktop and laptop, sharing files through CIFS/SAMBA and NFS. I was running out of storage space on the server, which only had a single 250GB 2.5” disk and I considered either to buy a cheap NAS box as the ReadyNAS Duo and have two disks in raid 1 (mirroring) or build a new server from scratch with room for several disks in raid 5.

Updates to the article

I have updated this post several times since I posted it and it now contains the following new sections:

Underpowered off-the-shelf NAS boxes

smallnetbuilder.com — write speed: around 15mb/s, read speed: 35MB/sec for files up to 128MB and less than 10MB/sec for larger files!.

Since I do image and video processing on large images on my desktop over NFS I need the speed to be better than this. So I started to look into more expensive NAS enclosures, but even though I think I would have been perfectly happy with a QNAP TS-459 Pro+ Turbo NAS enclosure performance wise, it is silly expensive (6000DKK = 1050 USD) without disks. After all this box is nothing but underpowered, single-purposed Linux servers with a webgui, surely I can do this better my self. Building my own server would also give me the power to run a HLDS server, saturate my gigabit network, running the LAMP stack, a squeezebox service and such.

Finding the right case

I started searching the web for a enclosure that was small (as in mini-itx), had a good build quality, supported more than two internal 3.5” sata drives (preferably hot-swappable) and finally somewhat silent cooling wise.

There isn’t many of those around, so I thought I had hit jackpot when I found Chenbro ES34069, which is quite ideal if it wasn’t because the number of compatible mainboards listed on the page is quite sparse and the few they listed was not available in Denmark. I contacted them to ask if some specific mainboards would fit and if they would update the page to cover more recent mini-itx boards, but after it took them 3 weeks to answer and at that time I have already moved on. For those interested their answer after three weeks was that the Intel DG45FC and DH57JG would fit their case.

Still searching the web I found a silentpcreview article which recommended a build based on a Lian Li PC-Q08B enclosure. This article is the inspiration for my build, modified to the availability in Denmark.

Lian Li PC-Q08B (739DKK = 129USD) – This Mini-itx enclosure has 6 internal 3.5” bays and is rated as very quite in several reviews. I can now add it is only really quite if you under voltage the fans a bit, but once this is done is very quiet.

Zotac G43-ITX (709DKK = 124USD) – This is a socket 775 mini-itx motherboard with 5 sata channels, e-sata and HDMI output. 6 sata channels would have been better, but 5 is good enough. The HDMI port makes for possible role as media server later on.

Core 2 Duo E7500 3 MB (960DKK = 170USD) – This was the cheapest Core 2 Duo socket 775 processor i could find. Since I had plans for HLDS server and perhaps even virtual machines I did not want a unpowered celeron.

CoolerMaster Silent Pro M500 (600DKK = 109USD) – This is a bit expensive power supply, but it should be silent according to online reviews.

Corsair 4GB DDR2 XMS2 PC6400 800MHZ (540DKK = 94USD) – I could do with 2GB, but the motherboard only has 2 slots, so I would rather upgrade now than later.

4 x Green WD10EARS (450DKK = 79USD per disk) – I wanted to make a raid 5, with one spare leaving me 2TB usable space. These disks should not use to much power and be reasonable quiet.

One 2.5” hard drive for the OS – I had an old 40GB SATA model I could use, but if I haven’t had this I would probably by a cheap 40GB SSD – just to avoid bottlenecking the server now when all the other components are so powerful.

The total price including the disks was 5700DKK (970USD) which is the same price as the QNAP TS-459 Pro+ Turbo NAS without any disks.

Choosing a specialized or normal NAS Linux distribution

I am quite experienced with Linux and am using CentOS or Debian for all my server needs. I do this even though I am aware of FreeNAS and Openfiler, but I prefer the flexibility of a distro which is not single purposed. I won’t to go details on how to install Linux machine, but I will say I used UNetbootin to boot the machine on a Debian squeeze netinstall image.

Setting up software raid on 4K sectors disks

First I did not think much about the fact that my new disk had 4K sectors (also called Advanced Format), but when I started to create my raid I quickly realized that I had done something wrong. My raid was building at 35MB/s which wasn’t too impressive. After having surfed google for a while I found that had partitioned my disks wrongly such that blocks were split over two physical sectors on the hard drive. I had used fdisk to partition my drives, but the default is apparently to partition in DOS-compatible mode, where the first sector is sector 63. Invoking fdisk with the options “-cu” will make fdisk default to non DOS-compatible mode and start a partition at sector 2048. After having formated using “fdisk -cu /dev/sda” I ended up with this:

After the 3-4 hours it took to initialize the raid I needed to create a file system on the raid. Again I need to align the filesystem with the disks, so I used the http://busybox.net/~aldot/mkfs_stride.html to calculate the best settings for my filesystem and the result was:

mkfs.ext4 -b4096-Estride=128,stripe-width=256/dev/md0

Recovering from a faulty disk

A important part of a RAID setup is the ability to cope with the failure of a faulty disk. Since I am using RAID5 + 1 spare I can afford one disk to die and the raid setup will start to rebuild on the spare disk giving me the time to order a new disk online. Since the enclosure I have chosen does not support hot-swap and the disk have no separate lights for each disk I need a way to find out which of the disks to replace. In the case that a disks has failed I would receive a mail from the mdadm daemon and by issuing the following command I could find out which disk has failed:

mdadm --detail/dev/md0

Assuming that /dev/sda has failed I can get the serial number of the disk using hdparm:

hdparm -i/dev/sda |grep SerialNo

and luckily the Western Digital disks I have came with a small sticker which shows the serial on the disk. Since I can’t do hot-swap I will then wait till the disks has finished rebuilding on the spare disk, power the server down, replace the faulty disk, power the server up again, partition the disks as above and finally add the new disk to the array with:

mdadm --add/dev/md0 /dev/sda1

Data scrubing

To ensure that a rebuild will run smoothly without any data reading errors it is commonly recommended to do a data scrub/check daily or weekly. To initiate a data scrub of the raid use the following command:

Using the sync_speed and sync_completed virtual files i created a small munin script to monitor the progress:
Once I have tweaked this a bit I might upload it to http://exchange.munin-monitoring.org/

A HTPC media server

The Zotac motherboard has a HDMI port and the HDMI output was immediately recognized on my Sony KDL-40W5500 LCD television. So now I have placed my server underneath my TV with a HDMI cable to the TV and a audio cable to my stereo.

The great thing about this build compared to a atom or arm based build or even a stand alone QNAP NAS is that I have more than plenty of performance to watch youtube videos, run grooveshark or watch HD video recorded on my digital camera. In addition I can now browse the web at get the full experience. Since this build has a dual core Intel E7500@2.93GHz, 4GB DDR3 ram and a onboard Intel GMA X4500 graphics processor the setup laughs at flash heavy pages and does everything and normal computer would. I am very happy.

Conclusion

I am very happy with my build. It is almost completely silent, quite powerful and cheaper than buying something comparable from the shelf.

Interesting article, thanks, this is something that I want to do and I have been testing FreeNAS 0.7 and 8.x and so far I like the 0.7 version because of the RAID ability (easy to set up) but most importantly, disk encryption. I would not want my NAS server stolen with all my data unencrypted. Since FreeNAS 8 currently doesn’t support encryption, I have been looking at alternatives to FreeNAS 8 and was thinking of going the Linux route, your article is interesting and informative. I might test your steps but add in disk encryption, will have to see if there is a method of mounting the encrypted volumes via a GUI, or I could probably set up SSH and use that to mount.

This is cool. I stumbled onto your site via the sshfs/automount article. Then I saw this article about a NAS/HTPC and thought “Hey, that’s what I’m building”. And then you’re using the exact same q08 chassis I’m using! Really reassuring to see you completed the build and it was quiet:) By the way, what are you using to undervolt your fans? My MB has one controlled fan power output, but I guess I’ll need to get something for the exaust fan as well…
For the insides I’ve got a core i5 2500k, 8gb of ddr3 1600mhz and an ASUS P8H67-I motherboard. The board has a SATA 6gb/s connector so I’m going to “have to” get a fast ssd to make use of that later on, but I’ll start of with an old harddrive. I’m still waiting for my PSU, so I haven’t started by build yet. I also can’t afford much storage right now on my students budget, but in a couple of months I’ll be checking back in here to get the details of your RAID setup. It looked sweet.

Sorry Mattias, your post got lost in my todo list. For undervolting the fans I used the classic Zalman Fanmate 2.

As you can read in my followup article: Rebuilding and updating my Linux NAS and HTPC server I had to rebuild my NAS again since it was hurt by lightning. The funny thing is that I now also decided on a 2500K – which works out great with Debian testing and kernel 3.0.1. XBMC is awesome on this setup.

Luckily the thunder only killed off the mainboard, so I could use the hard disks again.

My backup strategy consists of 2 measures:
1) Nightly backup via rsync of everything to like-minded Linux oriented friend’s server.
2) Monthly rsync-based backup to external hard drive – I am using rsnapshot for this. This way I have 6 archives, but since it uses hard links it doesn’t take up more place than the diff of the files.

I am glad that you like and that you contribute to it as well with our conversation. At first my setup was 3 disk RAID-5 with one spare and only later I grew it to a 4 disk RAID-6, so probably I would have chosen differently now, but also using the http://busybox.net/~aldot/mkfs_stride.html . It is a interesting read, but reformatting again seems to much hassle since I already am able to saturate my gigabit network as it is which is my measure of success at the moment. I was to redo the build again – 64K seems as the right choice, but it is not really bad with my 512K chunks.

Regarding bitmaps I remember I investigated it, but I think I forgot about it, so I have enabled it now. I also remember that I tuned my speed_limit_min, speed_limit_max and stripe_cache_size to get better performance at some point:

Since I realized I didn’t correctly aligned my RAID (4K issue) and I still have my external USB drives with all my data I decided to try, but boy I don’t know how to make that script work…
so I guess I’ll settle for the conclusion of 64k. But I really would like to run the test… I’ll try a bit longer!
What tool do you use to assess the speed of your RAID?
I did a check after reading it on your blog (echo check >> /sys/block/md0/md/sync_action) and the speed was about 17-20 MB and it took about 24 hours and this is what sprung me to action. 😉

anyway when I was only looking at /proc/mdstat I tried to run a tuning script (which I rewrote from an existing one) and it gave me 10% better performances.
if you are interested it’s here and you can run it in an “informative” way, no actual system modification get executed:http://ubuntuforums.org/showpost.php?p=11622143&postcount=22

Davide: I currently do not use any privacy enforcing methods as I have complete trust in this friend (which is also family) and do not have anything secret beyond normal privacy. We have however tried out the tool called http://duplicity.nongnu.org/:

Duplicity backs directories by producing encrypted tar-format volumes and uploading them to a remote or local file server. Because duplicity uses librsync, the incremental archives are space efficient and only record the parts of files that have changed since the last backup. Because duplicity uses GnuPG to encrypt and/or sign these archives, they will be safe from spying and/or modification by the server.

The disadvantage from this tool however is that recovering files is more cumbersome.

Hi Thomas,
it’s very interesting that you found that value for stripe_cache_size to be optimal. I wrote a script to find sub-optimal settings and I got the same result.
Give it a go if you have time, I’d like to hear if it works:

good news all around! my video driver issue has been fixed (so now I can use 1920x1080i and 1280x720p over HDMI with no problem) and after aligning partitions, using the right chunk size and tuning some parameters, scrubbing the md device got 10 times faster!!!

Thanks for posting the details to your build. I too was looking at some of the pre-built appliances , but this route sounds much better. This will be the way I will do as well. Now to start looking for parts!

I have had stability problems with win 7 and my latest WD 3TB drive so I am tempted to switch over to centos. Though I recently updated the bios and it hasn’t crashed since so I’m sitting on the fence for now. It’d be especially nice to have ssh access and to seamlessly install git, subversion, and all of the free apps on apache for sharing files without dealing with the mess of doing that on win 7.

The space under my stairs has a solid-core door with gaskets and a separate air intake through the lowest two stairs and a temperature controlled exhaust fan that parallels my dryer duct, so noise isn’t a huge concern. Sometimes I sleep in that space (when guests are in my room) and then I power the server down for light as much as fan noise.)

The biggest problem with the zotac board is that it only has 3 sata ports, so I can’t max out the 4 drives even if I wanted to. Performance seems fine for netflix and decoding local avi’s or listening to music.

I’m curious if you have any thoughts on what you would do in my shoes. Here are the options I see:

1a. Jetway JNF99fl-525-lf (6 sata, atom d525) to take over the 1160 +
1b. Travla T1200 for the zotac board which will serve up htpc and perhaps secondary web/file server and maybe eventually convert the jetway to freenas and have a 3rd board for apache/git/etc but this seems overkill

2a. Jetway board as above for files
2b. Travla 2u enclosure for both the zotac and jetway boards
(not sure what I do with the 1u enclosure with this plan)

3a. The zotac board you are using or some other higher performance board (seems like I don’t really need the performance though)
3b. This plan necessitates the 2u enclosure for clearance above the fan/heatshield

4. Jetway board, move the zotac to a client pc elsewhere in the house (but most of my client pc’s are mac’s now since the hardware is so much nicer.

I am not really current on what is hot since I did my build, but I can give my more personal experiences. As you probably have read I am using my build as a HTPC, but I am currently considering a fanless graphics cards so I can have the HDMI link also serve audio. The current motherboard I have used in the build doesn’t support audio through the HDMI.

Regarding the builds I guess I would try to consolidate as much as I could on one machine – this would save time on administration and electricity. My build replaced two servers. Even though I think you are right that the dual core atom processor is probably capable of doing some transcoding and heavier stuff I would make a more build with a little more juice. One benefit of a more powerful processor would be that you could create a basic Linux installation running as hypervisor and create multiple servers on the same machine if you need the multitude.

Hmm, this was a bit of a confusing answer, but what I meant to say was that I would go for a single powerful setup rather than several lower performance nodes. Also, remember to check the HDMI capabilities of the board or consider the space for graphics card.

Those HDDs have no TLER (Time Limited Error Recovery), and as such it’s easy for one drive to be dropped from the RAID array. If this happen the spare drive recovery process would put extra stress on the remaining drives, increasing the chances of another drive failure, with dramatic consequences.

It happen to me with my QNAP 409 when upgrading in RAID 5 from 1TB disks to 2TB disks.

I’ve read RAID 1 is not affected by the TLER’s lack because it has no parity calculation, even if I’m not sure about the rationale.

Copy pasted from WD site (via another thread):

When an error is found on a desktop edition hard drive, the drive will enter into a deep recovery cycle to attempt to repair the error, recover the data from the problematic area, and then reallocate a dedicated area to replace the problematic area. This process can take up to 2 minutes depending on the severity of the issue. Most RAID controllers allow a very short amount of time for a hard drive to recover from an error. If a hard drive takes too long to complete this process, the drive will be dropped from the RAID array. Most RAID controllers allow from 7 to 15 seconds for error recovery before dropping a hard drive from an array. Western Digital does not recommend installing desktop edition hard drives in an enterprise environment (on a RAID controller).

Western Digital RAID edition hard drives have a feature called TLER (Time Limited Error Recovery) which stops the hard drive from entering into a deep recovery cycle. The hard drive will only spend 7 seconds to attempt to recover. This means that the hard drive will not be dropped from a RAID array. Though TLER is designed for RAID environments, it is fully compatible and will not be detrimental when used in non-RAID environments.

Ferdinando Ametrano: I see your concern and I have tried to find a answer for my self for the last few hours. As far as I now understand TLER is a matter to be concerned with when using hardware raid, but using software raid this is not actually a problem:

An editor on the site smallnetbuilder.com has queried the manufacturers of consumer NAS boxes that has non-TLER drives on their recommended list of drives about TLER. The answer was the following:

The responses I received from Synology, QNAP, NETGEAR and Buffalo all indicated that their NAS RAID controllers don’t depend on or even listen to TLER, CCTL, ERC or any other similar error recovery signal from their drives. Instead, their software RAID controllers have their own criteria for drive timeouts, retries and when a drive is finally marked bad.
…
So is there any benefit to using TLER / CCTL / ERC drives? Maybe. These features usually come on “Enterprise” grade drives (WD Caviar RE series, Seagate Barracuda ES, ES.2, Samsung Spinpoint F1), which are built to take the constant, hard use of business environments. So investing in these more expensive drives is probably a smart move if your NAS is under constant heavy use. But it will be the more robust drive construction and not TLER / CCTL / ERC that will make your RAID NAS more reliable.

1st of all, I use an SSD to run the OS on, and created a raid5 set on my non SSD-disks containing the storage, /tmp and /var, without creating RAID enabled partitions.
That way I don’t encounter your dos 4k sector issue
But if I’d do it again, I wouldn’t go for the SSD solution anymore…
my stripe_cache_size is set to 4096, as it was better than 8192
Here are my speed results:

I did, however, find some issues with the WD green disks. I saw my Load_Cycle_Count go up too fast.
Apparently WD Green disks have an issue (with Linux). The heads are parked way too frequently, causing your Load_Cycle_Count to skyrocket.
I found some details on the issue at hand here
It boils down to this:
for dev in /dev/sd[a-d]; do smartctl -a $dev | grep Load_Cycle_Count; done;

If you want your drives to live for about 4 years, you’ll have to stay below 9. (considering the drives have a max lifetime of 300,000 Load_Cycle_Count, some models have twice that much)
after 210 hours, I had an average of 50 Loop cycle counts per hour. My drives were only going to last for about 8 months!

After fiddling about a bit with different settings, I gave idle3-tools a try. Although not really stable, it did fix my issue.
My Load_Cycle_Count want down from about 50 per hour to about 0 (yay!)
The idle3-tools iterface is a bit buggy, so you’ll have to set the value, and then verify it. Rince and repeat until you get to about 30 seconds. I used 2400 to get 25.2 seconds. Don’t disable it. Apparently it doesn’t solve the issue (I haven’t ventured into this yet, as my SAN is in production)
Ah, and don’t forget: power-cycle your NAS after setting this. A mere reboot doesn’t apply the new hdd firmware settings.

Is this setup still what you would go for? I am considering doing a very similar thing now.
But does it work well to run everything in one OS with XBMC running on top as the frontend? I mean, there is a lot of stuff going on in the background, does it never interfere with XBMC?

I was thinking about running the server as a virtual Linux guest in a Linux host, where the host really only has the (VirtualBox) image and XBMC running.. Perhaps I am only making life more difficult for myself?
Hypervisor is another approach, but again likely more complex.

The new build is quite powerful and has no problem running the Mate desktop (Gnome branch) behind XBMC – BD images run fine in full HD over HDMI, so I quite happy.

In regards to virtualization is definitely possible, but maybe not really a good way to fare if you do not have a specific reason to do so. Performance wise I see no reason to do so, but if you have some very specific security, networking or testing reason I can’t see why not to try so.

I would say that Debian testing is preferred. Especially in the new i2500k based build as the built in GPU benefits greatly from the advances in the very recent kernels.

I didn’t really consider Xeon’s as I didn’t need the power an the have no built-in GPU. The i5 2500K is even maybe to powerful for the build, but I like the possibilities of growth in features my server can do. I haven’t chosen ECC ram either since it only servers my self and a few others and some downtime is acceptable – that said I have never had a problem which I would related to memory issues and my uptime is usually months only abrupted by kernel upgrades.

I don’t consider it all that bad – this is while having 5 disks spinning and running XBMC on a TV over HDMI. The Pico PSU I have seen from a quick google search didn’t have the possibility of powering 5 disks. But overall you are right – it could probably have been made more power efficient, but my emphasis while building the box was to be powerfull and silent.

Thanks for such a great guide. I have tried FreeNAS, did not like it. I am testing OpenMediaVault now (Debian based appliance) and it is fine but I need more flexibillity and I want to include an HTPC part like you did. Your project is what I need ! I will do it on a rescued Lenovo Desktop with a P4 HT, 2 Gb of RAM, a Nvidia GT210 video card with HDMI, one IDE 40 Gb as the system drive and 2 WD Green 2TB disks for the data (music, movies, photos). If do my project with Ubuntu Desktop 12.04 instead of Debian, will I be able to replicate your setup ? And if I need to minimize the disks spinning and power consumption, would that be better to use rsnapshot (http://www.rsnapshot.org) instead of RAID-1 for the data disks ? Thanks in advance !

I am glad you liked it. It is isn’t too demanding to run the NAS and the XMBC part for that mater especially since you have a dedicated GPU. The only reason I chose Debian (beside beeing annoyed with the new GUI’s of Ubuntu) is that the kernel was the most reason one that supported my i5-2500K on die GPU. Since this is not your problem Ubuntu should be just fine.

The disks spinning down is only depending on your usage, so in case the idle at night the will spin down – I have actually written about this in this very article See /etc/hdparm.conf. Finally even though I am great fan and user of rsnapshot I would choose RAID 1 over rsnapshot, both since you do not have to have downtime when one of the disks eventually dies and the reading speed would be higher (not that it likely matters).

Great tutorial. exactly what I was looking for.
I thought that I can contribute a little information. I did the installation as documented here, but I failed a few times due to freezing of the OS ( Debian wheezy with kernel 3.2). Annoyed I dis some research and apparently kernels 3.2 and older cannot handle properly the new Ivy bridge that comes with the newer 3 generation CPUs from Intel.
I did the upgrade to a 3.6 kernel and to my surprise the rebuild speed rose from 75KB/s to 128KB/s with no other changes than the new kernel.

I heave a question regarding RAID5. Is it really necessary to create a spare or can you build a raid 5 with no spare and keep the spare disk on the shelf and replace it only when it is necessary. To my the hot spare looks more like a luxury item, especially when you don’t have hot plugs.

Thanks – I am glad you liked it. It sounds great with the with the 3.6 kernel even though I think you mean 125MB/s instead of 128KB/s which would be a little slow for rebuilding:) I haven’t experienced any freezing of my i5-2500K sandy, so it is, as you say, only ivy bridge that this problem exists. When I did my build I had to use 3.0 (or 3.1) as Sandy Bridge was not supported on 2.6.

There is no problem in creating a RAID 5 without hot spares, but the idea of the hot spare is to have the raid rebuild from a critical state as fast a possible. If you have it on shelf anyways I would definitely set it up as a hot spare or alternatively create a RAID 6 instead as I did. With a RAID 6 you have 2 parity disks and can sustain 2 disk failures without data loss – which is safer as your data is not as vulnerable if you a single disk failure. You can see how I did in my continuation of this article: http://www.tjansson.dk/?p=1660

Sorry for my mistake, it is MB not KB.
On another not I have a question about the effective size of a RAID array. I did a search on the Internet and there are various calculators that give different answers, but all estimate more than I have.

I have five WD 3TB red setup in RAID 5+1. I used your settings from above and I get a reported effective size by df -h of 6TB. Most estimates from these calculators give much more.
I know that the effective size of a disk is actually 2.7 and the effective total size would be around 10.8TB. If I consider the size of taken by the parity (size of on disk) it should still be somewhere around 8.1 TB, but I get 6TB instead.
Do you have any good calculation methods to estimate the size?

Below is the information from mdadm and from fdisk for one of the HDD.

It seems as if you have partitioned your disks wrongly since the use space per disk is around 2TB and not the 2.7TB as you wrote. At least one of them was partioned wrongly. If you have not already started to fill them up I would just redo it all. Otherwise I guess you could take them out of the raid on at the time, repartition them and add them to slowly have all at 2.7TB and then grow the raid volume. The easiest and quickest way is however is just to redo it.

I am glad to hear that it is working. I think the reason that it is working is that you are not using any partitioning when you write

/dev/sd{b,c,d,e,f}

instead of

/dev/sd{b,c,d,e,f}1

So you are accessing the whole devices. This can be fine – I have done similarly when I had to format 3TB drives on old Linux installations.

The only caveat could come up if you need to rebuild the raid in another machine. I don’t think Linux automatically recognizes the full disk as a raid member automatically as there are no partition with the usual flags. The is not a terrible hard problem – you would just have to gather the raid again more specifically by telling mdadm which disks are in the raid.

The reason I mention it is just that I was very happily surprised by Linux to automatically assembled my RAID when I had to move my disks to another computer.