Dropsafe

by Alec Muffett

Hybrid Workstation / ZFS NAS System Built!

Simon kindly lent me his expertise and we assembled my new AMD Solaris box; the final configuration for those of you who are interested in such things, was:

  • 2.1GHz dual-core AMD Athlon X2 BE-2350 @ 45W
  • Zalman CP9500 CPU Cooler
  • Thermaltake Toughpower 1000W PSU
  • Thermaltake Armor (full-tower with side fan)
  • 4x1Gb Corsair Dominator DDR2 non-ECC SDRAM
  • Asus M2N32 WS-Professional Mobo
  • LiteOn LH-20A1H-11C 2x DVD-RW
  • 160Gb Hitachi Deskstar PATA 7K160 (“root” disk)
  • 5x500Gb Samsung Spin-Point NCQ SATA-II (raid-z array)

Total expenditure: (updated: 20 August 07) £1059; the choices may seem a little odd to some, but here’s the thinking:

  • The 160Gb root-disk is single-platter in the hope of long life, though it is nominally a scratch disk
  • The 5x 500Gb disks are cheap, nominally quiet, and will provide 2Tb of storage in a single ZFS zpool
  • The 4x 1Gb memory are more expensive than usual, but all matched and with heatsinks
  • Since Solaris can’t run AMD PowerNow on current multi-core AMD chips, the CPU is high efficiency 45W model to reduce heat
  • The motherboard has 9 SATA ports.
  • The case can easily accomodate another 3+ disks, so in a couple of years I can add 3x 1Tb spindles and make a 4Tb RAID-Z

The big disks will be pooled into a single RAID-Z1, and then I propose multiple filestores will be exported via NFS with differing qualities of service per-mountpoint:

    basic – your “vanilla” filesystem
    compressed – stuff you want to squash (gzip)
    redundant – stuff you are worried about losing (ditto2)
    archival – stuff you absolutely cannot lose (ditto3)

I’m not proposing to mix compression with redundancy for the same reason I never used to compress system dumps before they hit tape; it just seems wrong to mix the two policies. Regular snapshots to be taken via cron.

Nevada testing starts tomorrow. I’ll update you on progress.

⊞

Comments

17 responses to “Hybrid Workstation / ZFS NAS System Built!”

  1. Stephen Usher

    That looks like a useful server spec.

    As for archival backup a colleague in another department has implemented a hierachical, archival back-up system using rsync with hard links, somehow diff’ing the previous archive with the current filesystem state. i.e. each file is only stored once on the disk but is referenced in every archival copy of the directory tree where it appears, one directory tree per back-up instance.

    I know that you can have archival snapshots under ZFS but that can only be stored on the original filesystem and not on a remote server.

    Reply
  2. alecm

    >I know that you can have archival snapshots under ZFS
    >but that can only be stored on the original filesystem
    >and not on a remote server.

    Errrrrrrrrrrrrrrrrrrrrrrrrrrrrrr…

    Lemme see:
    remote fileserver uses ZFS, exports files via NFS.
    cronjob on server takes snapshot of MOUNTPOINT@COOKIE
    client nfs-mounts SERVER:MOUNTPOINT on /FOO
    client sees /FOO/.zfs/snapshot/COOKIE/… readonly snapshot.

    Where’s the problem? You want remote invocation of snapshot-taking? Use SSH and RBAC.

    Reply
  3. Stephen Usher

    It’s more the duplicate back-up of data on a second set of disks (and server) at a separate location. i.e. locational redundancy as well as data storage redundancy.

    Reply
  4. alecm

    >It’s more the duplicate back-up of data on a second set of disks
    >(and server) at a separate location. i.e. locational redundancy
    >as well as data storage redundancy

    Ah, you’re taling about having a mirror elsewhere, for disaster recovery? Yeah, I can see that’d be a bind, but then that’s not what ZFS is really about.

    Although on the other hand I can imagine my colleagues telling me to create a mirror of local disks and iSCSI LUNs…

    …but it would probably be better to deal with this at the fibrechannel/wires and spindles level.

    Reply
  5. Stephen Usher

    Yeah, I know.. but as a back-up solution it’s pretty nifty (using rsync creatively).

    Maybe I’m just warey of having all my data in one array (or even two) after the disaster with multiple disk failures 18 months ago. (25 out of 32 disks dying within 2-3 weeks in two arrays, one the off-line mirror of the other.)

    Reply
  6. Bhima

    Have you been able to do any testing on this? Does Solaris see all 10 the SATA II ports? (so both the 6 that are hooked up to nForce 590 SLI chipset and the 4 attached to the Marvell 88E614x). I understand there is a ZFS or RAID-Z test suite, have you run this? How about the GigE (Marvell 88E1116) ports, does Solaris recognize both of these?

    Lastly, I don’t really need a SLI board, do you have any idea if the current version of Solaris Express would support other ASUSTek boards using these chipsets?

    I’ve just about given up trying to find an authoritative source about which current motherboards and which SATA & Network chips are supported. I’ve seen a lot of comments like “should work” but very little of “I tried it and it worked for me”. I don’t really feel comfortable making a purchase based on a comment that it “should work”.

    Reply
  7. alecm

    Okay! Dealing with Bhima’s questions in approximate order:

    1) I have plumbed up 5 of the main SATA ports, and they work fine using Nevada build 68

    2) I don’t know about the rest of the SATA ports since I only have 5 SATA devices

    3) I don’t know where the RAIDZ test suite is, but I ran Bonnie++ on it somewhat haphazardly; the figures I got suggested I was getting typically 1Gb/s throughput, +/- 300Mb/s – ie: 700..1300Mb/s

    4) I was delighted to find the ethernet ports were detected as “nge0” and “nge1” out of the box, with no configuration required. I can confirm they run at Gigabit speeds.

    5) I’m not using SLI either, and am too dim to guess about other boards. The reason I have the Pro card is because of the massive number of SATA ports. Once the price of Terabyte spindles drops, I intend to add another 3 or 4 of them. I am assuming the other ports work.

    6) There is some disturbing notice on boot about:


    out of I/O resources on bridge: bus 0x3, dev 0x0, func 0x1, for secondary bus 0x5
    devices under bridge bus 0x3, dev 0x0, func 0x1 will not be assigned I/O ports

    …but I don’t know what that means, yet.

    7) I sympathise. The same worry bit me. That said, I had the same fear about buying a Linux machine in 1994 (bought a Dell 466/ME as a result) and look where we are now with Linux… 🙂

    Reply
  8. Bhima

    This is mostly good news!

    However, being that there are 6 ports on NVIDIA chipset it is possible that you have all your devices plugged into those ports and those ports serviced by the Marvell chip are not recognized. Do you have access to a spare SATA device you could test? I really, really would be appreciative!

    also do you have speakers attached? does the sound work?

    I hear what you’re saying about the Linux comparison but when tried Linux I was using an old PC rather than buying a brand new multi Tb server. So I really hope I don’t have to wait 15 years for Solaris to catch up hardware support wise!

    One more thing… I’m interested in hearing more about this “qualities of service” via NFS mount point scheme you mentioned.

    Anyway thanks for the info, I’m looking forward to moving away from the 4 hard drives duct-taped on the side of my PowerMac.

    Reply
  9. Bhima

    Perhaps you could run the “Sun Device Detection Tool” located here: http://www.sun.com/bigadmin/hcl/hcts/device_detect.html

    This would likely answer all my questions and tell you what exactly that error message is.

    Thanks!
    Bhima

    Reply
  10. Chris Samuel

    I would have thought there was always the option of running half the devices on one controller and half on the other..

    Reply
  11. Mark Ashley

    Not mirroring your root drive Alec? 🙂

    Reply
  12. alecm

    Nah – compared to the 2Tb RAID, the 160Gb is trivial, and will mostly be Solaris, SunStudio, a couple of LiveUpgrade partitions and a whole lot of scratch space.

    Reply
  13. David Smith

    Very cool. I’m looking forward to your results. Thank you for publishing the equipment list. I was thinking of doing something very similar and this equipment lists helps a great deal. I use ZFS at work, and I want the same for home to protect my data. Are you adding a graphics adapter, or are you going to configure this system from the serial port or install over the network?

    Reply
  14. alecm

    Hi David – I’ve got an nVidia graphics card, I forget the model but I think it’s the Quadro FX 1400 — not being one for gaming it’s just a fairly basic thing, does not need its own PSU to operate, but it is capable of driving the 1920 x 1200 / WUXGA Dell 2407WFP I’ve bought.

    Reply
  15. alecm

    Well, the Quadro FX 1400 video card is by far the single nosiest component in the box; so I am going to have to replace it with something passively cooled.

    Am starting to play with the noise levels, by plugging in various fans.

    Definitely use the speed controller for the Zalman fan, it’s a lot quieter at low revs.

    The side fan is very quiet – but so large that the noise from inside escapes.

    I’m considering a fan-controller for the 4 accessible fans.

    Suggestions, anyone?

    Reply
  16. Srini

    Hi,

    Did you have any other problems with this mother till now, Have you upgraded to any latest solaris builds.

    Thanks,
    Srini.

    Reply
  17. Srini

    Sorry its not MOTHER but MOTHERBOARD OR MAINBOARD..

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

More posts