Time for an upgrade!
I’ve been running a Dell PowerVault 705N for about the last 4 years now as my home file server. It’s run flawlessly and has yet to drop a drive. It originally shipped with (4) 60GB Quantum hard drives. I picked it up used for next to nothing and upgraded to (4) 160GB Western Digital drives. After setting it up I came to the realization that the 705N (which is really a rebranded Snap Server 4100) had the LBA hardware limitation that limits the maximum usable size of each drive to 137GB. So my 160GB drives quickly became 137GB drives. Not a huge deal but it would have been nice to use all the space on the drives. The 705N was setup with RAID5 giving a total of about 400GB of redundant storage. After 4 years of building up my mp3 and Divx collection I’m coming very close to running out of space on it, so I decided it was time for and upgrade.
Buy or build?
I looked around on the web and eBay and found that the amount of NAS and NAS-like devices available has increased dramatically over the past 5 years. I thought about upgrading to a newer Dell PowerVault like the 715N or 725N, but did not like that they were no longer rebranded Snap Servers (which are BSD based) and were now Windows based running some version of Windows Server or Data Center. Also, the Dell and Microsoft branding seem to keep the cost and resale value rather high. They still fetch about $800-$1200 on eBay with only 250GB drives installed. I also looked into some new NAS servers like the Dell NF1000 and the Snap Server 410. But soon was discouraged by the price tag. I also looked into some of the “cheaper” NAS servers like the Thecus N5200 and the Buffalo TeraStation, but they still seemed too pricy for the amount of storage and features you get. I came to the conclusion that I was going to have to build this thing myself.
Features
Some of the features I was looking for in my NAS are:
* Support RAID5 (either hardware of software based)
I’ve always liked the performance and fault tolerance of RAID5.
* Capable of running 4 or more SATA II hard drives
You need at least 3 for RAID5 and I don’t like odd numbers, so 4 it is!
* Gigabit connectivity
For better performance over the network when copying large files (like vob’s and iso’s).
* Fairly fast 64bit CPU and at least 1GB of memory
It seems many of the low-end NAS devices suffer from under powered CPUs and not enough memory.
* 64bit Operating System
32bit is so last century!
* Snapshots
Once I saw snapshots in action on a NetApp I knew I had to have it.
* Mutli-platform support
I run a mix of Windows, UNIX, and Linux systems.
* Upgradable
In case I want to increase the memory and hard drive sizes in the future.
OS? Windows…We don’t need no stinkin’ Windows!
First off there is no way I’m entrusting all of my data to a Windows based OS, so that’s out of the question. Before purchasing any hardware I spent quite a few months looking into to some of the more popular NAS *nix distros such as FreeNAS, OpenFiler, and DSSLite. These are all free and are either FreeBSD or Linux based OS’s. They are all specifically targeted for NAS appliances. They all seemed to have most of the same features and over all they have pretty good functionality. Since the sole purpose of these distros are to provide NAS they are somewhat limited to what else they can do. They all have web-based GUI interfaces to manage their setup and that’s about it. There’s no easy way to add functionality or other applications to them.
What flavor of *NIX would you like?
I’ve been working with UNIX for almost 10 years now so I am very familiar with it and have great confidence in its performance and stability. If my data has to be stored somewhere safe it’s going on some type of *NIX system (whether it be UNIX, BSD, or some flavor of Linux). I’ve had experience with most of the major flavors of UNIX, including Solaris, HP-UX, AIX, and IRIX. With the exception of Solaris (which offers x86/x64 versions) all of these OS’s only run on proprietary (and typically expensive) hardware. Because of this their market share seems to have dried up over the past 5 years or so. SGI has discontinued all UNIX based workstations and has also announced the end of life for IRIX in 2013. HP has stopped marketing UNIX based workstations and all servers, with the exception of enterprise level servers. HP-UX popularity is also declining and may end up going the way of IRIX. This leaves Solaris and AIX as the two most popular UNIX operating systems. I happen to be a UNIX admin and I support these two platforms at my job, so either is fine by me. Since AIX does not offer support for x86 and x64 architectures, this leaves Solaris as the UNIX of choice for my NAS.
Enter Solaris 10 and the ZFS filesystem
I’ve been working with Solaris since version 7 and have seen it mature into its current release of Solaris 10 8/07. You can’t beat the price of this OS (free!) and it has very up to date patch support and a fairly large user community. With the introduction of the ZFS filesystem Sun has put itself in an ideal position to compete in the NAS market. Many others have already realized this and built their own NAS servers for home use. Also, some companies are building their own NAS OS’s based off OpenSolaris (i.e. NexentaStor). There are many users in the OpenSolaris, Sun, and ArsTechnica forums that have already built their own NAS servers. In fact many of these people’s websites and blogs helped convince me that this is what I wanted to do as well. Here are a few links to some of them:
Recent home project: ZFS NAS server
http://thnetos.wordpress.com/2007/07/02/recent-home-project-zfs-nas-server/
Hybrid Workstation / ZFS NAS System Built!
http://www.crypticide.com/dropsafe/article/2091
New home server
http://blogs.sun.com/barts/
Getting Started with Solaris 10 and ZFS
http://www.dodd.org/blog/2007/03/getting-started-with-solaris-10-and-zfs/
I’m sure there are many more out there but these are some of the ones that inspired me.
“If you build it they will come”
Time to start buying some new hardware. I started with the thought of matching the current footprint of my Dell PowerVault 705N which is a 1U rack mount case. After some further research the 1U chassis is pretty limited as to what you can put in it. It also poses some problems with which components will actually fit in it. I decided to settle on a 2U chassis instead. With the exception of a special size power supply most PC components will fit without any problems. I opted for a chassis that had four 5.25” bays in the front so that I could install some nice removable and lockable hard drive bays. Next was to pick a motherboard that had good Solaris 10 hardware support. After reading some websites and blogs, I opted for the ASUS M2NPV-VM which is an AMD AM2 based board. Pretty much all of the onboard devices have out of the box hardware support for Solaris 10. I think the only things that are not usable are the TV and HDTV outputs. Others have reported problems with the onboard sound card but it works fine for me (maybe this was fixed in Solaris 10 8/07). This board also has many of the features I was looking for, it can run four SATA II hard drives, has Gigabit Ethernet, and supports some of the fastest 64bit CPU’s as well as supports up to 8GB of DDR2 800 memory. Next up was CPU. I selected the Athlon64 X2 BE-2350. It’s a fairly cheap dual-core 2.1GHz CPU. As an added benefit it is also a low power processor using only 45W. This is a good thing for a system that is meant to run 24/7/365. Next I selected two G.Skill 1GB DDR2 800 DIMMS. This left 2 memory slots open for future expansion. Next up power supply. I had to pick a power supply that was designed to fit in a 2U chassis as well as have a 24pin main power connector and 6pin 12V connector to meet the ATX12V standard the ASUS motherboard requires. I selected a 400W 2U Startech model. Next a SATA RAID card with at least four ports. After searching around for cards that were supported on Solaris 10, I quickly realized that these things are expensive. I then came across this document (link) from Sun that outlines how well ZFS with raidz handles RAID on its own. Furthermore, hardware RAID controllers are no longer necessary and can actually lower performance on ZFS filesystems. That being said I decided that I was going to skip buying this expensive component. Next thing I bought was some 5.25” removable and lockable hard drive bays. I picked out the IcyDock MB122SKGF-B which has some nice removable and lockable trays. They also have a blue LCD display that shows HDD temperature, fan status, and drive usage time. They also have audible alarms for fan status and drive temperature. The next item I needed was some kind of drive to install the OS on. I want to keep the OS on a separate drive from the data disks. I decided that I wanted to put the OS on a drive that had no moving parts, so basically some type of flash based drive. I looked into CompactFlash as well as some USB thumbdrives. Both of these could probably work but would involve some special attention to when installing Solaris. I then came across the IDE flash modules made by Transcend. They come in sizes from 32MB to 8GB. I settled on a 4GB one that will hold almost a complete Solaris 10 install (End-user bundle). The cool thing about these devices is that the OS and BIOS see them as a regular IDE hard drive, this way no special setup is required. They also have low power consumption and are very small in size. The last thing I needed to buy was the hard drives. I selected the 500GB Seagate ST3500630NS. These drives are the enterprise series of drives that come with a 5 year warranty and are geared towards RAID usage.
| 2U Rackmount Case with 4 x 5.25” drive bays |  |
$37 + shipping |
| ASUS M2NPV-VM AM2 NVIDIA Motherboard |  |
$89 + shipping |
| AMD Athlon 64 AM2 X2 BE-2350 45W Processor CPU |  |
$67 + shipping |
| G.SKILL 2GB (2 x 1GB) DDR2 800 (PC2 6400) Memory |  |
$54 + shipping |
| Startech 2U Rackmount 400W Power Supply |  |
$82 + shipping |
| (4) Icy Dock MB122SKGF-B SATA Removable Hard Drive Mobile Rack with LCD Display |  |
$168 + shipping |
| Transcend 4GB IDE Flash Module |  |
$90 + shipping |
| (4) Seagate ST3500630NS 500GB SATA Hard Drives |  |
$360 + shipping |
| AeroCool EasyWatch Fan Controller |  |
$25 + shipping |
| Total | Total = $972 + shipping costs |
Here are some pictures of the completed build.
Once all of the hardware was together and working fine, it was time to install Solaris 10 x86 8/07. The installation process is pretty simple and can be done from either 5 CD’s or 1 DVD. I had to rig up a temporary CD-ROM drive borrowed from my old Ultra 10. The process is similar to most Linux installs, basically setup hostname, network configuration, root password, disk partitioning, and which packages to install. The only part that involved some work was picking the packages. Because the Transcend IDE flash module is only 4GB, I could not fit the entire End-user bundle. I had to manually go through and pick out packages that I knew I would not need. Most of them were unnecessary hardware drivers and also some applications that I didn’t need, the biggest one being Star Office. Also, when partitioning the disk I removed the swap slice so that Solaris would not use any swap on the flash module. Hopefully this will decrease the amount of writes to the flash drive and increase its lifespan. These flash based disks are limited to the amount of writes/rewrites that can be performed (although it is rated at 2,000,000 program/erase cycles). After an hour or two the Solaris installation was complete and I was presented with the login screen. The next thing I did was download the latest recommended patch bundle and install it. After a reboot from the patch bundle I set up the Sun Update Manager to provide notifications of new patches. Once this was setup I had about 40 additional patches ready for install. After this the server was fully patched and up to date. Next it was time to setup ZFS on the four 500GB drives. I could have set up the zpool through the command line but I noticed that there is a new ZFS GUI interface that is now included on the Solaris install. It is accessed through the Sun Java Web Console, which needs to be enabled before you can use it (svcadm enable webconsole). Once the console is enabled you can access it through https://localhost:6789. The ZFS GUI is very well designed and shows all aspects of your ZFS storage pools, filesystems, volumes, snapshots, etc. It also has many built-in wizards to guide you through some of the most common ZFS tasks. At the end of each wizard it presents you with the actual command line input it will use to run the command (kind of reminds me of SMIT in AIX). I think the ZFS GUI is a nice feature especially for anyone that hates the CLI (you know who you are!). Also, it provides a nice way to view the ZFS configuration and status from other computers over a secure web connection. This gives me some similar functionality as the Dell PowerVault tools that came on my 705N. Here are some screen shots of setting up ZFS in the GUI.
Logging in to the Java Web Console
Web Console main page
ZFS Administration page
Creating a storage pool
Assigning disks to the pool
Command that will be run to create the pool
Command completion
Basic ZFS information
Storage pool status
Once ZFS is setup the next thing to configure is SAMBA. SAMBA comes with Solaris 10 and it is as simple as editing the /etc/sfw/smb.conf file and enabling it (svcadm enable samba wins). I also created a dedicated user to own all data stored in my share. Once this is done the share is accessible to Windows though Explorer or by mapping a drive to it. At this point I can access the share from any of my computers though either NFS or SAMBA.
Move that data
Now
that the new NAS server was ready it was time to start copying over
some data. I temporarily mounted my existing share from the
old
PowerVault and started copying the data over. I let the
copying go on
for hours and eventually had all of the data on the new
server. I
monitored the hard drive temps to see what my maximum operating
temperature was in order to determine if I had enough
cooling. During
the long copy process, two of the drives closest to the power supply
peaked at about 110° F. After searching around on the web it
looks
like the optimum operating temperature for hard drives is somewhere
around 80-90° F. Turns out the original layout of my 2U
chassis has
the power supply moved inboard about 6 inches into the case.
This
didn’t leave enough breathing room for the two drives on the right side
of the case. I did some slight case-modding and moved the
power supply
to the rear of the case where it belongs. This helped bring
all four
drives into acceptable operating temperature. The top two
drives run
at about 88° F and the bottom two run at about 91° F.
Snapshots
galore!
Now
that I had ZFS raidz functional I needed to setup automated snapshot
creation. I wanted to have hourly, nightly, and monthly
snapshots
automatically created. I knew that this was going to require
a shell
script running from a cron job. I stumbled upon a very nice
script to
do pretty much what I wanted here...
http://blogs.sun.com/chrisg/entry/snapping_every_minute . I
slightly
modified this script because I definitely did not need snapshots every
minute. Here’s a copy of my script
ZFSsnapshot.ksh
Keeping
an eye on things
In
addition to the snapshot script I wrote a few other scripts to help
keep the ZFS filesystem in good shape and to alert me should something
go wrong. The first of these scripts is a ZFS scrubbing
script. This
script will perform a scrub on the specified ZFS filesystem and send a
text message to my cell phone via Google’s Send To Phone
service. The
second script performs a basic ZFS health check and will also send a
text to my cell phone if something is wrong.
ZFSscrub.ksh
ZFShealth.ksh
sendtext.sh
Backing up the system
In
order to keep a backup of the system drive I wrote a script to perform
a flar create to make a complete backup of the system in case of a
failure of the system drive. The script creates a flash
archive that
could be used to restore the system.
create_image.ksh
After less than a year of running my new NAS with four 500GB drives, I’ve already run out of room. The time had come to upgrade to 1TB drives. The nice thing is that the 1TB drives don’t cost much more than I spent on my 500GB drives. For a while I had thought about getting a different case and running 8 drives. However, that would mean buying not only 4 new drives, but also a new case, drive trays, and a 4 port SATAII card. Add to that the additional power usage and noise associated with the extra drives. I searched around the net and tried to find out if I could grow the RAIDZ pool in ZFS. Most of the info I found stated that this was currently not supported, and there was no easy way to do this other than backing up all the data somewhere else and then copying it to the new pool of drives. This would require that I have all 8 drives at once as well as somewhere to put about 1.5TB of data temporarily. I also found a few stories of people swapping out each of the smaller drives with a larger one and letting the pool resilver. However, I was still getting mixed information about whether this worked or not. Well I decided to start swapping the drives one at a time. I bought the 1TB drives one at a time off eBay (about $125 each) and as I replaced the drives I would let the new drive run for a week or two just to make sure it was in good shape. I would then sell each of the 500GB drives on eBay for about $40 each. So my overall cost was right around $340 (not bad for doubling my capacity). During the time I spent waiting for the last 1TB drive to arrive, I thought “what if this doesn’t work?” Well, if it didn’t my last resort would be to buy a 1.5TB backup drive (probably a Western Digital MyBook) to backup my data in order to transfer to the new pool. Well, that made me think “why not try this virtually?” I had played with both VMware and Sun’s VirtualBox before and they were pretty easy to setup and test with. I downloaded VirtualBox and the latest version of OpenSolaris (2008.11). I had been pondering switching to OpenSolaris after seeing a couple of its newest features (such as Time Slider, and the new Package Manager). Also, I was hoping that I might be able to take advantage of newer SATA drivers, as I have been running my drives in IDE legacy mode. The setup took maybe an hour and I was able to start some testing. I started with a 4GB virtual system disk to represent my current 4GB flash module. I then added four 4GB SATA drives to quickly represent my 500GB drives. I then set the RAIDZ to match my current settings. Then one by one I replaced the virtual 4GB drives with new 8GB virtual drives to represent my 1TB drives. I replaced the last drive and crossed my fingers hoping the pool would now have twice the capacity. Well it didn’t, at least not at first. I had read somewhere that exporting/importing the pool might do the trick. I ran a quick pool export and import and sure enough it did work! The capacity had now doubled. This gave me some pretty good hope that when my last 1TB drive came in I would be good to go. Below is a screenshot of me completing the drive swap and then exporting and importing the zpool. Note that before the export/import the available disk space was only 78GB but after it is now 1.41TB. NICE!!
Migrating to OpenSolaris 2008.11
Solaris 10 8/07 has been running rock solid on my NAS for almost a year now. However with some of the recent features added to OpenSolaris, such as Time Slider, new Package Manager, remote desktop through web browser, as well as many new drivers, I figured it was time for an OS change.
Time Slider-
Automatically snapshot your personal files at regular intervals with the ZFS file system. Once snapshots have been taken, use the Time Slider feature in the file manager to browse changes to your files over time, view them and recover changes. Here's a quick shot of it in action. At one point I had a file called TEST, on the "live" filesystem it was deleted and now there are two new files in it's place. All we have to do is move the little slider to a past point in time and we have read-only access to the old file.
Package Manager-
The Package Manager has undergone a significant level of improvements, with support for adding new package authorities, management of boot environments allowing you to quickly activate and switch on package upgrade, improved search and performance, and better classification of packages.
Transmission (Torrent client)-
Transmission is an easy-to-use BitTorrent client that's now available for OpenSolaris. Open your torrent file and start your download!
GNOME Remote Desktop-
Access your desktop from a web browser on another computer.
There is one particular feature missing from OpenSolaris 2008.11, the ZFS GUI. I've really grown to like the ZFS GUI. I still do some things from the command line, but the GUI really shows all of the important ZFS info in an easy to use interface. I noticed that the ZFS GUI did come with the latest version of SolarisExpress. I figured that it shouldn't be too hard to rip the packages out of the SolarisExpress DVD and install them on OpenSolaris. The following packages need to be copied from the SolarisExpress DVD in order to get the Java Webconsole and ZFS GUI working on OpenSolaris...
SUNWmcon SUNWmconr SUNWmcos SUNWmcosx
SUNWsmcon SUNWzfsgr SUNWzfsgu SUNWj5rt
SUNWj5dev SUNWmctag SUNWjhrt SUNWjhdev
SUNWmfrun SUNWdtcor SUNWctpls SUNWjato
I just kept trying to install the packages until the order was finally right. Once you get them installed run svcadm enable webconsole, and then reboot. You'll also have to login with a user other than root (root is now a role not an actual user). You will also need to grant yourself the proper roles/profiles (System > Administration > Users and Groups). Once I did that I was able to successfully login. I did notice that the RBAC seems a bit buggy in OpenSolaris; basically every time you edit users through the GUI it wipes out whatever you had set in the /etc/user_attr file. So instead of even touching the GUI, I've just set my profile and roles right in the /etc/user_attr manually. Everything seems to work fine now.
Snippet from /etc/user_attr file...
rob:::::auths=solaris.*;roles=root;type=normal;profiles;=Web Console Management,ZFS File System Management,ZFS Storage Management
If you can't stand not being able to log in as root, you can run the following command to convert root back to a normal user account instead of just a role.
# rolemod -K type=normal root
Questions or comments? Contact me at robhensel@hotmail.com