Backing up bits

:: Monday, July 26th, 2004 @ 10:55:34 am

:: Tags: Computers

There are more than 113,000 files in my top-level photo directory (which thankfully now resides on a RAID 5 box in the closet).

Data loss terrifies me, and so I have finally have started the arduous process of archiving everything onto DVD-R. I do own a 33GB tape drive, but at $60 or more per tape, I’d rather use 7 DVDs (less than $5) to store the same amount of information.

In the immediate future dual-layer writeable DVDs will be available in quantity and presumably will begin to fall in price, and in the longer term Blu-Ray will up the storage capacities of a single disc to 25/50GB. But at some point you have to commit to a media format and start backing up.

The problem is that the rate at which my data needs is growing is surpassing the rate at which cost-effective removeable media capacities are growing. For example, it would be great to be able to back up my life onto 20 or so discs, but currently I would probably require 100 discs or more. So instead, I tend to keep my data on multiple drives in separate machines, reserving the risk of catastrophic data loss for acts of God (and more recently, the risk of viruses brought behind the firewall by roommates). That’s not good enough, I don’t think. I need to have copies stored off-site — hence, the DVD-burning binge that I’m going through right now.

I’ve thought about rsyncing my data to a remote server, but even at an upload speed of 100 KB/sec, 500GB would take roughly 57 days to upload. I suppose I could walk into a CoLo with a server already seeded with the data, banking on the fact that incremental updates won’t take up as much bandwidth…

Also, I’m amazed at how many of my friends simply do not back their data up. As much as possible, I’ve been forcing loved-ones to use mirrored hard disks in their desktop machines, which will at least account for the possibility of drive failure.

Blah blah blah. Boring content today. But it’s what is on my mind. :)

| Popularity: 7% | San Francisco, CA | link | trackback | Jul 26, 2004 10:55:34

:: 9 comments (rss)

1. posted by syndromes on Mon, July 26, 2004 @ 2:21 pm

   Well, consider your job partially done - your entries have spurred me to create a dedicated backup server. I’ll just be doing raid 1, but better than what I (don’t) do now.
   
   I’ll also be getting a DL-DVD burner, but i’m not sure that’ll i’ll actually be dilligent in backing them up there as well. I’m lazy ya know :)

2. posted by guest on Mon, July 26, 2004 @ 4:52 pm

   i am also terrified at data loss with my digital photos. i recently had a hard drive crash on me and long story short - i’m also burning my archive to dvd-r.
   
   i’ve noticed, on occasion, if i do a binary comparison of the images i’ve burned compared to their source, some get corrupted and i’ll have re-burn the disk. what tool do you use, if any, for binary comparisons? i’ve used winmerge, but i’ve noticed that it’s not always accurate.

3. posted by echeng on Mon, July 26, 2004 @ 4:56 pm

   hey, guest. i use the “verify” checkbox in Nero. never had it fail. although interestingly, with one drive, i consistely get failures when it tries to finalize a DVD-R. works fine from my thinkpad’s multi-drive, though.

4. posted by Alex on Mon, July 26, 2004 @ 6:38 pm

   I had that finalize DVD failures with my nec dvdr. I just bought a plextor that “seems” to work fine, but i still haven’t used it enough to be sure.
   
   I should work out some kind of mirroring for myself, at least on one of my drives (i only have ~ 100GB of important data :) but it’s growing faster now that i have a dslr

5. posted by vinh on Tue, July 27, 2004 @ 9:05 am

   :) “Data loss terrifies me” from the same guy that disabled the diskhead parking safety feature on his laptop.
   
   the DVDs are just like seeding the a remote server over a low-bandwidth line, once you’ve gotten past the initial set of burning, so long as you continually burn data incrementally, it won’t be so bad. it’s only bad now because you hadn’t taken any backups until now, so this one is a full backup.
   
   take the time to catalog these DVDs now while you’re burning… otherwise, once you’ve amassed a library it’s gonna be a PITA to find anything.

6. posted by echeng on Tue, July 27, 2004 @ 9:19 am

   do you have a suggestion for a file cataloging program? i’m not interested in cataloging my local volumes. i only want to catalog the DVDs i’m burning…

7. posted by vinh on Tue, July 27, 2004 @ 8:48 pm

   =\ unfortunately no… i’ve been meaning to either find or come up with something myself for a long time now… i archive so much stuff onto CDs, but can never remember what CD it’s on…
   
   in your case, whatever program you find or script out yourself, assuming that you’re already using some kind of asset library/manager, you should look for something that can pull the descriptions or captions you have for your images… the idea being that you’d want something where you can search for a term and pull up a list of files and what CD/DVDs they’re on.
   
   and if you’re too lazy to look for something, keep an excel spreadsheet or access database with a CD/DVD name and a description of its contents. :)

8. posted by stephane on Sun, August 01, 2004 @ 9:15 am

   FWIW you can see the equipment that Bjørn Rørslett is using. Gee, those pros are scary.

9. posted by stephane on Sun, August 01, 2004 @ 9:17 am

   looks like b2 does not support utf-8 so read that encoded data as ‘Bjorn Rorslett’

:: leave a reply