ENTRIES
Welcome to Eric Cheng's online journal! You are not logged in. [ Log in ]
«  :: index ::  »

Time Machine vs Crashplan and other thoughts on backup

:: Monday, August 13th, 2012 @ 8:55:12 am

:: Tags: , ,

Those of you who follow me on social media know that I complain often about the work required to keep my data securely and reliably backed up. Since 1997, I’ve been a hardcore Crashplan user. I love that it features continuous, versioned backups to multiple destinations, and that it supports multiple platforms. You can backup any Mac, Windows or Linux machine to virtually any destination (folder, second computer, or cloud).

I continue to recommend Crashplan over Time Machine for normal users; for a very-small monthy subscription fee, you can back up every machine in your family to an external drive, and to the cloud—a fast, local backup plus a secure offsite backup, all in one solution.

Unfortunately, I am not a typical user. I have about 10 terabytes of data that I need to keep easily accessible and securely backed up. About half of that are static image files, which are stored on a redundant NAS volume and backed up offsite, which leaves the other half—about 5 TB—needing to be actively backed up. To seed 5 TB to a Crashplan destination on my gigabit LAN, it takes between 7-10 days, which yields an average backup speed of 6-8 MB/s. That is really, really slow.1 Still, Crashplan mostly just worked, and I have used it in the past to retrieve old versions of files that had somehow become corrupted. The problem was that when Crashplan didn’t work, it was often catastrophic. After months of continuous backups of huge datasets, Crashplan inevitably started to do bad things. Typical problems included:

  1. Neverending synchronization. Crashplan was left in “Synchronizing…” state for hours or days at a time.
  2. Connection problems. Crashplan would stop being able to connect to a destination, even though a machine right next to the problem machine would connect and backup without problems.

Each time, I submitted support tickets that were answered within about 48 hours, and each time, I was left without a working backup. In order to recover from these bad states, I had to start a new backup, leaving me in a vulnerable state during the time necessary to seed a new backup (7-10 days).

After 3-4 instances of this, I have come to the connclusion Crashplan is simply not reliable for large datasets. I think that if I had less than 1 TB of data, I would still happily be using Crashplan, but it stops working way too often when datasets are too large.And so, I recently gave Apple’s Time Machine a try again, pointing it at a Synology NAS.

I was surprised by how fast Time Machine seeded its backup, and was pleased that Time Machine didn’t simply crash when I launched it, which is what happened the first time I tried it (when the feature was new). The ~5 TB backup took less than 1.5 days, which means that it averaged nearly 40 MB/s—totally respectable for moving almost 5 million files across the network. However, after about a week, Time Machine told me that it wanted to create a new backup “to improve reliability,” which is dumbed-down speak for having detected corrupted sparse bundle. This would have left me in a state without a backup again—precisely what I want to avoid. A Google search yields many reports of problems using Time Machine to backup to NAS boxes that have built-in Time Machine support.

As a final attempt to make this work, I am now using Time Machine to backup to an iSCSI volume on a Synology NAS. Synology’s wiki suggests using iSCSI “if the network connection will be regularly interrupted,” and pretty much every modern machine falls into this category, since notebooks are closed and moved, and desktop machines regularly go to sleep or are turned off. So we’ll see what happens with Time Machine over iSCSI.

If my most recent effort doesn’t work, I’ll likely try to decrease the size of the data that needs to be actively backed up by switching photo and video backups to cron’ed, incremental rsync to a NAS volume. I assume that both Crashplan and Time Machine do better when dealing with smaller amounts of data.

I know there is a better way. NAS boxes seem to backup to other NAS boxes very well (e.g., Synology to Synology via rsync), and working with RAW image files off of gigabit is totally fine (I can barely tell that they aren’t local). I envision a setup in which my entire working image library is on NAS, backed up to a second local NAS and to a third offsite NAS. All of my video files would be on a Thunderbolt-attached RAID, which would be backed up to NAS using rsync. My boot disk, Applications, general data and project files would be backed up using Time Machine. I think it could work, but I’d have to be strict about compartmentalizing data into source files that never change (RAW image and video files), files that are constantly in flux that need to be continuously backed up (libraries, project files and other working files).


  1. Note that seeding 5 TB to the cloud on my ~180 KB/s ADSL2 connection would take nearly a year. 

| San Francisco, CA | link | trackback | Aug 13, 2012 08:55:12
  • Andy

    Eric, I revisited Time Machine last night on a whim for Leslie’s Macbook Air, and after 2 years of using CrashPlan I was pleasantly surprised at how fast it backed up 19GB of data to an external drive. I am not brave enough to back up over the air to my Synology box, based on reports I have also read about with reliability issues OTA. I am investigating dropping CrashPlan and moving to a 3xNAS box approach like you are mentioning, but I am concerned about speed over ethernet for Lightroom work with my raw files on the NAS box. Another concern is rsync’ing from the office to home and it’s bandwidth requirements. I could always seed the first rsync (and any larger ones after a big data increase like a safari) by bringing NAS #3 from home to the office, but my main concern is actually going over the 250GB / month Comcast data limit. Just thinking out loud. You and I have similar data sets, but mine is more like 6TB and changes more often.

  • Leon

    I have 8.3TB to backup and it say it will take 3 years to backup. I get around 650Kbps day or night.

  • Rob

    I’d hate to have any single point of failure, and I also see different benefits of local AND off-site backups. For this reason, I use Time Machine (with a connected hard drive) and Crashplan. It’s not like it has to be one or the other. I even add to that cloning of my drive every few week, and move a cloned drive offsite once-in-awhile. Yes, I’m super-paranoid.

  • Tyler

    Eric, I was wondering if you could provide an update on your degree of success using Time Machine to backup to an iSCI volume on your Synology. I have about 1TB to backup to a Synology NAS and thus far am having the same problems you’ve described using both Time Machine and CrashPlan.

  • http://echeng.com/ Eric Cheng

    Hey, Tyler. I’ve stopped using iSCSI, so I, unfortunately, cannot help. I’ve been using Crashplan for my most-important 1TB of data, and do manual, incremental backups using ZFS.

ARCHIVES
Journal Home
Where is Eric? (password)
Stuff for Sale
February 2014 (2)
December 2013 (1)
October 2013 (1)
June 2013 (3)
May 2013 (2)
April 2013 (3)
March 2013 (1)
February 2013 (2)
January 2013 (3)
November 2012 (2)
October 2012 (3)
September 2012 (8)
August 2012 (8)
July 2012 (8)
June 2012 (8)
May 2012 (5)
April 2012 (8)
March 2012 (15)
February 2012 (7)
January 2012 (6)
December 2011 (8)
November 2011 (10)
October 2011 (12)
September 2011 (8)
August 2011 (14)
July 2011 (9)
June 2011 (9)
May 2011 (11)
April 2011 (11)
March 2011 (12)
February 2011 (23)
January 2011 (22)
December 2010 (16)
November 2010 (17)
October 2010 (26)
September 2010 (24)
August 2010 (24)
July 2010 (30)
June 2010 (26)
May 2010 (21)
April 2010 (26)
March 2010 (19)
February 2010 (17)
January 2010 (29)
December 2009 (21)
November 2009 (23)
October 2009 (32)
September 2009 (19)
August 2009 (34)
July 2009 (21)
June 2009 (30)
May 2009 (23)
April 2009 (18)
March 2009 (6)
February 2009 (25)
January 2009 (5)
December 2008 (6)
November 2008 (22)
October 2008 (27)
September 2008 (25)
August 2008 (34)
July 2008 (34)
June 2008 (32)
May 2008 (26)
April 2008 (15)
March 2008 (19)
February 2008 (31)
January 2008 (43)
December 2007 (33)
November 2007 (29)
October 2007 (29)
September 2007 (9)
August 2007 (19)
July 2007 (10)
June 2007 (17)
May 2007 (26)
April 2007 (38)
March 2007 (39)
February 2007 (13)
January 2007 (35)
December 2006 (35)
November 2006 (14)
October 2006 (6)
September 2006 (20)
August 2006 (24)
July 2006 (32)
June 2006 (17)
May 2006 (23)
April 2006 (16)
March 2006 (16)
February 2006 (26)
January 2006 (33)
December 2005 (17)
November 2005 (21)
October 2005 (18)
September 2005 (17)
August 2005 (5)
July 2005 (15)
June 2005 (20)
May 2005 (25)
April 2005 (7)
March 2005 (22)
February 2005 (20)
January 2005 (38)
December 2004 (6)
November 2004 (24)
October 2004 (16)
September 2004 (22)
August 2004 (12)
July 2004 (17)
June 2004 (15)
May 2004 (11)
April 2004 (35)
March 2004 (40)
February 2004 (29)
January 2004 (36)
December 2003 (20)
November 2003 (18)
October 2003 (10)
September 2003 (18)
August 2003 (10)
July 2003 (34)
June 2003 (12)
May 2003 (49)
April 2003 (42)
March 2003 (42)
February 2003 (15)
January 2003 (7)
December 2002 (17)
November 2002 (19)
October 2002 (24)
September 2002 (22)
August 2002 (20)
July 2002 (21)
June 2002 (14)
May 2002 (15)
April 2002 (11)
March 2002 (13)
February 2002 (20)
January 2002 (17)
December 2001 (16)
Even Older Journal
Travel Journals

CATEGORIES / TAGS
(25) (2) (1) (3) (1) (1) (1) (6) (2) (3) (11) (8) (3) (1) (1) (4) (2) (4) (2) (1) (6) (1) (1) (1) (6) (2) (1) (1) (1) (3) (1) (5) (1) (1) (23) (1) (1) (1) (1) (1) (14) (1) (10) (1) (1) (2) (1) (1) (1) (27) (6) (3) (2) (4) (4) (1) (1) (41) (11) (12) (4) (38) (1) (3) (2) (4) (1) (1) (1) (1) (2) (1) (1) (1) (1) (1) (10) (25) (8) (3) (2) (3) (2) (1) (5) (1) (1) (2) (1) (1) (14) (1) (5) (1) (1) (5) (43) (1) (1) (1) (3) (24) (1) (1) (1) (1) (5) (1) (4) (1) (1) (10) (1) (3) (1) (1) (1) (1) (6) (5) (1) (1) (1) (3) (1) (3) (1) (1) (1) (69) (4) (3) (7) (3) (1) (16) (6) (1) (29) (1) (7) (1) (4) (4) (4) (1) (1) (1) (1) (1) (1) (1) (10) (4) (4) (2) (1) (89) (14) (1) (2) (79) (2) (2) (1) (1) (1) (1) (1) (1) (3) (2) (3) (1) (1) (24) (3) (5) (4) (1) (2) (1)
MOST POPULAR
Most Popular Posts of All Time


Eric Cheng's RSS Journal Journal RSS
Eric Cheng's RSS Journal Comments RSS

proudly powered by wordpress
script exec time: 0.50s
i hate computers.