otherthoughts:
The active and archived jobs are indeed stored on the same file system. The large logical volume (currently 4 terabytes using a hard drive manufacturers' definition of a terabyte) that is provided to the operating system by the RAID controller has a single partition on it that occupies the entire volume. This partition has, in the root of its file system, a directory for active jobs, and also a directory for archives. From a network client, a single samba share is mounted, and because the root of the file system is shared, the client sees a "jobs" directory, an "archive" directory, and several others.
On most file systems, each directory has its own index, which lists files and directories inside it and points to their locations. When you are navigating the directory tree, each directory points to the next. Some file systems may have a master index, but it is the linked directory indexes that are sequentially accessed as the directory hierarchy is traversed.
As an analogy, consider a printed volume of the US Internal Revenue Code. But instead of the clauses being logically arranged in order by chapter then part, section, etc., they are arranged in a seemingly random order, with every clause appended to the end in the order that they were created, and divided among multiple pages when they are amended with more words. This is how file systems typically grow, because an insertion that leaves everything neatly ordered would require a prohibitively expensive shift of all the data that occurs after the insertion. This isn't a perfect analogy, because with file systems there is such a thing as "deletion." If you had an enormous table of contents at the beginning that listed all of the clauses in the apparently random order, with references to what page they were on, then the worst-case scenario is that you would have to read the entire table to find a single clause when you knew what chapter, part, etc. it was in. Instead, imagine that the first page is a table of contents that lists only the chapters with page numbers. Knowing the full path of the clause in question, you would quickly find it, even though the list is not in any logical order. You would then go to that page, where you would find a table for all of the parts of that chapter. You would then go to the page for the part, then section, sub-section, etc. until you found the page with the clause you're looking for. This is how most common file systems work, and is the reason why many file systems have a limit to the number of items nested within a single directory (because a fixed amount of space is allocated for the directory's index). You can therefore increase the number of files and directories by a power of two (e.g., from 1,000 to 1,000,000) while only doubling the time it takes for the operating system to find a single file or directory. In a well-divided system this could mean the difference between one hundredth of a second and two hundredths of a second. The operating system, network clients, etc., never have a clue what's in a given directory unless they specifically list its contents - it may as well be unused space.
If you have OS X, you can see evidence of this behavior by comparing the time it takes to do each of the following:
1. navigate to /Library/Printers/PPDs/Contents/Resorces/en.lproj.
2. Open a finder window an go to the root of the system drive. It should list the directories Applications, Library, System and Users. Change the view mode to list view if it isn't already (command-2). Alt-click on the disclosure arrow to the left of the Library directory. It should now list the entire contents of the Library directory and all of its nested directories.
If there was a single master index that had to be read in its entirety for every listing of a directory, the second operation would be quicker than the first (only one read instead of many). The first operation should in fact be faster because you are taking one focused path through the directory tree, while the second operation has to bounce up and down touching every branch at every level of the tree.
pcmodem:
I couldn't imagine having several thousands of optical disks - that sounds massive! There will be an off-site backup as soon as the clone is installed at our other facility, so our only single point of failure will be a comet.