Back up And Archiving ....ideas

[kyle]

The only question I still ponder is how you structure your directory hierarchy containing tens of thousands of jobs so that you avoid such bulky directories?

Our job "numbers" are a letter followed by four numbers, e.g., A0000-A9999, B0000-B9999, etc. The archive directory has directories named A0, A1, A2, etc., and A0 contains all jobs that begin with those two characters. So the greatest number of possible directories in a single index would be 1000, which isn't too many to list fairly quickly. The perfect division of a large number of items in terms of balancing the directory tree and capping the number of items in a single directory would be to take the square root of the quantity and put that many items in each sub-directory, which would yield the same number of sub-directories as items within each. But if that's too many, take the cube root of the number and create one extra level, etc.

 

[DonBlock]

Pete, you are more on the right track for today's technology.

We set up our customers who really want secure data in a similar manner. With storage being relatively cheap these days, we set up a NAS RAID 5, usually 2TB, with a 2TB mirrored drive connected to it via Firewire 800. The customer has the option to do a permanent archive from the RAID to either tape or our recommendation is to DVD. As you stated, this is very fast and all of your information is always available. Very few have more than 2TB online storage requirement.

Your talking less than $2,000.00 for the RAID 5 and less than $1,000.00 for the backup 2TB. And these systems come with the software for scheduling and performing the backups...

At this price, some have even decided to get 2, 2TB backup drives and swap them out daily to secure the information off site.

Also forget about the compression onto tapes. In out industry most files will not compress from the original size so look for enough pure on-line storage and backup solution.

 

[otherthoughts]

Our job "numbers" are a letter followed by four numbers, e.g., A0000-A9999, B0000-B9999, etc. The archive directory has directories named A0, A1, A2, etc., and A0 contains all jobs that begin with those two characters. So the greatest number of possible directories in a single index would be 1000, which isn't too many to list fairly quickly. The perfect division of a large number of items in terms of balancing the directory tree and capping the number of items in a single directory would be to take the square root of the quantity and put that many items in each sub-directory, which would yield the same number of sub-directories as items within each. But if that's too many, take the cube root of the number and create one extra level, etc.

Beautiful logic and explanation Kyle!

Thanks again for sharing your time and skill with us!
Otherthoughts

 

[Drewstre]

Hiya-
At our shop, we too have made the switch from DVD-based backup to hard drive archiving. This is used in conjunction with CD Finder to catalog the archive. To save space, we make a compressed .dmg out of each job folder using DMGConverter (it's free), and save those dmg's to two 1TB drives. Use CD Finder to catalog. While this gives us two copies, we are still looking into offsite solutions to complement our current scheme.

One interesting and very cool thing: the latest versions of CD Finder will catalog the contents of .dmg archives. Save space, retain searchability. Groovy.

Hope that helps,
Drewstre

 

[chiefy]

Archive and Backup priorities

Archive and Backup priorities

Archive:
I 'Archive' my old jobs to DLT4 (800gig) tapes via an archive app in my prepress software onto a dell lto library. Currently about 20,000 jobs. Its run to two tapes automatically and as the tapes fill i remove the copy off site. This seems to work well and any of my staff can then retrieve old jobs for reprint without having to know anything about how it works, tape libraries etc.

Backup:
I wanted to 'Backup' my live data onto the same LTO library utelising some of the spare slots but the archive software disables RSM on window03. So at the moment the backup dumps onto another server partition. This doesnt keep me feeling great as the servers are all in the same rack and if i had a total disaster i am stuffed! I want to run a backup on the weekend and incrementtal each night onto the tape libraries and then remove each weeks cycle of tapes off site and rotate around over three cycles so i always have two good sets off site. In the event of a total disaster i loose at most a weeks data even in the event of a fire. The live server is 2TB but is only using about 500gig currently as i keep the archives upto to date.

Can anyone advice on what i can use to 'share' the LTO library in order to do this?

 

[mattbeals]

Petabytes on a budget: How to build cheap cloud storage | Backblaze Blog

Keep it all online until the file system chokes, or you're able to upgrade to ZFS...

 

[Miconnex]

After years of working in prepress and fighting with vendors on archiving to tape I've come up with a solution that might solve some of these issues. Here are a few key points on this new software:

- Archives to hard drive
- Indexes all filenames and Indesign paragraphs
- Creates and indexes thumbnails of all images
- Runs on all your macs and logs everythings.

Check it out: Archivery - The Archive Solution the Print Industry Has Been Waiting For

Feedback is always appreciated.

Dave Williams
Miconnex Business Services Inc.