6 terabytes in a Mac Pro
:: Thursday, August 30th, 2007 @ 3:40:38 pm
:: Tags: Computers
I nearly ran out of space in my Mac Pro the other day, and it was painful. I had 4 x 500GB drives in the machine with an additional 4 x 500GB attached in various enclosures connected via Firewire 800, Firewire 400, and USB 2.0. I also do backups onto 2 NAS boxes and a standalone server, which hold a total of just over 6 more terabytes. Backups over the network are pretty painful on the Mac; all backup apps I’ve tried so far are about 3x slower when copying over the network than are comparable programs running in the Windows environment. As a result, I’ve migrated to primarily doing local backups.
Anyway, so I nearly ran out of space on my main working drives, which is where most of my photos and video work is stored. Luckily, music, movies, and ripped tv shows are stored on a NAS box; otherwise, this would have happened long ago.
Originally, this was my internal Mac Pro drive and volume configuration:
A: 1 x 500GB System
B: 2 x 500GB RAID 0 (striped, 1TB accessible) Data 1
C: 1 x 500GB Data 2
B and C filled up, and I was starting to use A for image storage, which was not ideal. I hate storing data on my system partitions because it makes reinstalling everything too complicated.
So here’s what I did. I bought a MaxConnect for Mac Pro, which allows the installation of two SATA hard disks in the space normally occupied by a second optical drive. Here’s what the product looks like:
I ordered six Hitachi 1TB hard drives (enterprise versions, whatever that means). They claim a “target” of 1.2 million hours MTBF (137 years), but considering that 1 out of the 6 drives was DOA, I don’t believe them. Anyway, I installed the six drives and partitioned them as recommended by the Mac Pro Performance Journal. Disk Utility.app is a little buggy and required several reboots in order to partition and software RAID all the drives as suggested, but everything basically worked as described.
Note that this installation is easy if you’re used to building your own systems, but if you’re a Mac guy who has only opened up your system to admire its internal beauty, you may not want to attempt using the MaxConnect. Routing cables around inside a Mac Pro isn’t as easy as it could be!
Here’s my new partition and volume configuration:
A2: 20GB partition x 6 = 120GB RAID 0 (striped) fast partition for PS3/Final Cut scratch
B2: 65GB partition x 6 = 195GB RAID 1+0 (striped over mirrored) boot partition
C2: 850GB partition x 6 = 2.5TB RAID 1+0 (striped over mirrored) data partition
I’m now completely mirrored while still having a performance boost from the striping. I have a super-fast partition for scratch space and temporary work, and I can lose any one disk without losing real data. If I lose a disk, I have a 60% of surviving a second drive failure at the same time. Also, see a discussion about RAID X+Y vs RAID Y+X. It seems to make sense, and is why I chose RAID 1+0 over RAID 0+1.
Disk Utility.app is a mess!
Unfortunately, I only gained 43% in accessible disk space by adding 200% in actual disk space, but the mirroring is worth the extra cost since I’ve got my entire life’s worth of photographic work sitting on the drives.
Performance
My RAID buddies out there hate disk benchmarking utilities, but I don’t know of any other way to measure performance. I ran Xbench and got the following numbers:
| Sequential | Stock 250GB/7200 | 6-Drive RAID 1+0 | 6-Drive RAID 0 |
| Uncached Write [4k blocks] | 87.28 MB/s | 356.82 MB/s | 383.97 MB/s |
| Uncached Write [256k blocks] | 64.32 MB/s | 225.70 MB/s | 321.53 MB/s |
| Uncached Read [4k blocks] | 28.30 MB/s | 19.80 MB/s | 21.45 MB/s |
| Uncached Read [256k blocks] | 73.19 MB/s | 195.44 MB/s | 422.62 MB/s |
| Random | |||
| Uncached Write [4k blocks] | 1.24 MB/s | 4.15 MB/s | 14.25 MB/s |
| Uncached Write [256k blocks] | 23.67 MB/s | 84.80 MB/s | 141.05 MB/s |
| Uncached Read [4k blocks] | 0.63 MB/s | 2.04 MB/s | 4.88 MB/s |
| Uncached Read [256k blocks] | 25.52 MB/s | 48.63 MB/s | 84.27 MB/s |
Who knows if the numbers are accurate? I get slightly different results every time I run the thing.
In any case, I’m happy now and have just over a terabyte of free space available locally, which should be sufficient for the next year or two.
I also ordered a WiebeTech RTX400-SV, a four-bay external eSata system that allows drives to be inserted like removable disks. All the data travels on a single eSata cable thanks to Steel Vine (port-multiplied eSata) technology. I think this setup will work out just fine for the immediate future. Next time, I may go with an Apple xServe RAID wired via fiber to a remote closet. :)
By the way, I buy most of my computer hardware from NewEgg (affiliate link), which has never failed me. They have a simple online RMA program, so if anything arrives DOA or has a problem within the return period, returning it is easy.
Finally, here are two hints for the installation of the MaxConnect and the max_flow fan system for the Mac Pro:
- Be sure you adjust the screws on the bottom of the top bay in the Mac Pro properly. Otherwise, the MaxConnect frame will not slide all the way in. The front of the MaxConnect tray should be 15mm from the front of the front lip of the bay when it is installed in the right place. If it isn’t pushed in far enough, the DVD drive tray may snag when you try to open and close it.
- When mounting the max_flow fan for the lower memory riser card, mount it so it does not touch the bay’s metal frame. If it touches, you will hear an annoying, high-pitched resonance when the fans run.
Journal RSS 
Eric,
You might also want to consider storing long term backups in a remote location. This prevents you from losing everything should something happen in your primary location.
Cross campus remote storage via fiber isnt as far fetched as you might think although the number of people that do it for personal reasons is probably extremely low.
Check out Sun’s Thumper. You can stick plenty of disk in there.
Jauder – yes, I do that already. It’s part of the reason I bought the RTX400-SV. I can just stick a drive in, fill it up, and move it off-site.
I love the 6 drive configuration, I’m doing my own with 750’s, but I don’t like the Mac Pro Journal suggestions since the author offered no evidence that his configuration actually improved performance at all!
I’d like to see application benchmarks like FCP once they’re configured to properly use the Speed and Data partitions. I’m not convinced that the modest and occasional performance benefits of the Speed partition compensate for the hard partitioning and forced seeking that results. I could easily see overall performance being degraded; in fact I’d expect it. “Speed” partitions should be on dedicated, very fast drives.
I’m disappointed that RAID 10 on OS X doesn’t match RAID 0 read speeds. Perhaps it does and the benchmarks used simply can’t show it. Given sufficient command queuing (not automatic with desktop systems and benchmarks!), identical drive counts should yield similar read speeds for RAID 0 and 1 or 10 unless the programmers are stupid.
Ah cool!
That WiebeTech looks interesting. I would be curious to know how well it works over time. What kind of speeds are you getting writing to it?
UERs for HDDs have been flat for many years now and in the field, even for Enterprise class drives (vs. “consumer” level drives, which are one order of magnitude down), they are in the 8-20TB range. Personally, this is frightening for someone who does a lot of I/O and actually cares about their data. The proliferation of common formats used for pictures, music, and video tend to mask this VERY well. “Who cares if a bit or two flips in your abc.mpg file?”
Other media (flash and optical) which do not have the benefit of Enterprise lineage currently boast even more dismal statistics.
If things don’t start to change, this issue will come to the forefront a bit sooner than most folks realize. The sky is certainly not falling this year, or next, but given exponential consumption and capability trends….
Eric, during an “average week” (or whatever your granularity of uptime is), can you send me the amount of “data read” shown by Activity Monitor. It’s a global that doesn’t provide per device information, but it’s a good way to get a feel for where you currently stand.
Those of you who do a lot of I/O and do NOT see this as an issue, ask me for the CERN report or simply try it out for yourself. Create a file with a well known pattern, read it back, repeat. There are some fine details (throttling your I/O, etc.) but the workload is simple and common. Try it on your favorite media.
Chris – Thanks for your input. I’ll send you some data when I’m actually around for a full week and have been cranking away on images!
[...] the full article, with benchmarks, click here. If you want to buy some prints of his more famous photos, go to his new web gallery. Posted in [...]
RAID 1+0 is a good choice for your work. Most of your I/O are sequential. This will allow especially faster read I/O. Entire stripes will be able to be read in parallel using either copy of each chunk. At some point, you may want to think about going to a low cost SAN. This would be easier in a Windows environment as they now have relatively low cost “SAN in a Box”
Good lord! How did you hook up the eSATA cables without removing the plastic (fan) assembly?
Hey, Chris! It was a pain in the ass, and involved needle-nose pliers. It is way harder than it should be.
Hey Eric, thanks for the info. I’m thinking about installing 6 drives in my new mac pro. Is it still working out for you? Any problems? Also, I assume you used the odd sata ports for the other drives, correct? Thanks!