PDF file size limit?

[rich apollo]

I'm working on a project involving some very large document sizes. InDesign packages could contain from 6-700MB to over 3GB of data. That's a single page with 4 or 5 images. We're working these on a Mac Pro with 6 cores, 96GB RAM, and a 1TB SSD hard drive with InDD CS6. We're also working on an iMac with 4cores, 8GB and a 1TB hard drive with CC2014. When trying to export directly to PDF, InDesign fails with the message, "Failed to export PDF." At times InDD will crash, and it looks like virtual memory errors. Watching the Activity Monitor shows that InDD really isn't touching the RAM.

We can successfully write Postscript, but I'd really rather go directly to PDF. When writing Postscript, InDD ties up more memory. It's like the Postscript pipeline is just bigger.

Can anyone give me a reason that this would be failing? Where should I be looking? Can I tweak the system to handle this? Is postscript the right answer?

I guess a next step would be to try CC2014 on the bigger machine to see if a 64-bit app can take advantage of the horsepower.

 

[wonderings]

I had issues when pushing the variable data to far with the built in tools for indesign with CS6, it was trying to make some massive documents. When I upgraded to CC2014, Indesign became a native 64 bit app and handled the problems I was having with CS6 much faster... and well, it actually completed exporting the job rather then crashing. The job I was exporting was no where near the size yours is, but it was a file size that was a couple of gigs with 10,000 pages or so if memory serves me correctly. There was some big improvements performance wise with Indesign CC 2014, I would give that a go. The 64 bit definitely utilizes the power of your machine well over what CS6 did.

 

[Tom Hallinan]

Another approach that has helped in such situations is PDF-native OPI. The document is exported from ID with all images omitted, which is extremely fast. The file transfer to the server is also very fast. There the images are inserted by the OPI server. All PDF objects, transparencies, etc. are preserved. This also allows different versions to be automatically created (e.g. low-res, high-res, ICC color matched for RGB, CMYK, iPad, etc.). The entire process is much faster than using ID alone.

This solution is primarily used in cases such as yours, or in remote production workflows, where document designers utilize low-res images to save Internet file transfer time and bandwidth.

Tom Hallinan
HELIOS Software GmbH

 

[mwc]

I'm working on a project involving some very large document sizes. InDesign packages could contain from 6-700MB to over 3GB of data. That's a single page with 4 or 5 images. .

Just curious if altering the PDF preset you are using would help you (changing the image resampling numbers and/or compression settings)?

 

[Shawn]

Hi Rich
Have you tried turning off background processing when exporting the PDF?

I've had better luck processing large or extremely complicated files with it turned off.
http://indesignsecrets.com/a-new-workaround-for-that-pesky-background-export-issue.php

Shawn

 

[rich apollo]

I can get low-res PDFs out. I having trouble getting something suitable for proofing.

I had tried disabling asynchronous exports, but that didn't solve it.

Tried with CC2014 on the larger machine, but no dice. We're about convinced that that machine is not running properly, though. The write to disk speeds are not where they should be.

For the moment Postscript seems the only option. That's fine, it will work, but I'd sure like to have a better understanding of what is happening.

I think I will post this over to the Adobe forum, too.

 

[David Milisock]

Asking maybe what may be a stupid question, but what is the page dimension of this one page document? We regularly write large PDF files in the sign business in terms of file data but the 200" page size has been issues. Using PC's not MACs older Adobe products and CorelDraw X7.

 

[rich apollo]

We're below the 200" limit. I think the largest file will be about 180" wide.

 

[David Milisock]

We're below the 200" limit. I think the largest file will be about 180" wide.

Ok now what may be another stupid question. What is is size and resolution of placed images? ​

 

[Greg G]

Rich, have you tried turning compression and downsampling off when generating the PDF?
Turning compression and downsampling off would reduce the amount of processing as the image data would be written straight to the PDF.
Are the images overcomplex - many layers and effects? Simplying the image files will also reduce processor and memory load.
As wondering suggested, the 64bit version will definitely be more capable of processing larger docs.

 

[rich apollo]

Tried with a variety of down sampling and compression settings, including none. I can get low-resolution PDFs to write - down sampling to about 150 ppi, but nothing suitable for proofing or production.

The placed images vary in size, but all are quite large. There is a background image (a tiff without transparency) that is 183" by 16" at 500 ppi. other images are 800ppi.

 

[David Milisock]

Tried with a variety of down sampling and compression settings, including none. I can get low-resolution PDFs to write - down sampling to about 150 ppi, but nothing suitable for proofing or production.

The placed images vary in size, but all are quite large. There is a background image (a tiff without transparency) that is 183" by 16" at 500 ppi. other images are 800ppi.

At 185" how are you outputting? If you do the math you may be hitting a TIF limit and may not need the resolution depending on how you're going to print. We use 150 and even 100 dpi for ink jet work regularly without any degradation of the print ​

 

[Greg G]

183x16" at 500ppi would require saving as a .psb file. These can be "forced" into Indesign but are not directly compatible - this may be the problem.
I simulated your scenario and always got "Failed to export PDF" when trying to export a high resolution PDF. Strangely I could get a 150ppi PDF to export as well?!

 

[David Milisock]

183x16" at 500ppi would require saving as a .psb file. These can be "forced" into Indesign but are not directly compatible - this may be the problem.
I simulated your scenario and always got "Failed to export PDF" when trying to export a high resolution PDF. Strangely I could get a 150ppi PDF to export as well?!

Being in the sign business I understand this ​completely this is a limit that we work with al the time. This high res PDF has 2 chances of working no way and no how.

 

[dabob]

Rich, maybe I'm missing something here . . if your outputting to an inkjet you shouldn't need 500 dpi or higher resolution files - why don't you resample them and then place them at like 150-175 dpi and give that a try we regularly get life size Point of Sales original photos - 6' tall 300 dpi images of a sports star they want to use at about 3 inches tall - the resulting effective dpi is in the 4000 - 5000 dpi range - sometimes I just have get them down to a reasonable size, 300 dpi at 100% and replace them .. . .

 

[David Milisock]

Rich, maybe I'm missing something here . . if your outputting to an inkjet you shouldn't need 500 dpi or higher resolution files - why don't you resample them and then place them at like 150-175 dpi and give that a try we regularly get life size Point of Sales original photos - 6' tall 300 dpi images of a sports star they want to use at about 3 inches tall - the resulting effective dpi is in the 4000 - 5000 dpi range - sometimes I just have get them down to a reasonable size, 300 dpi at 100% and replace them .. . .

I teach the four word principle, The Completion Backwards Principle. In other words build the file for it's intended destination, we do high res trade show booth backdrops and never need more than 125 DPI at placed size. You can see if the guy has or has not shaved the morning of the photoshoot viewed at 6".

 

[dabob]

I teach the four word principle, The Completion Backwards Principle. In other words build the file for it's intended destination, we do high res trade show booth backdrops and never need more than 125 DPI at placed size. You can see if the guy has or has not shaved the morning of the photoshoot viewed at 6".

David . . . that's more of less what I was saying . . why oh why are your images such a high resolution - get them to the rez you need at 100% - maybe a little higher but not 4 times or more than what you need . . . .

we're a primarily a litho shop hence the need for 300 dpi images

 

[mojoprime]

a couple of hardware thoughts. how full is the drive on the mac pro? the swap files might be choking it, especially if you say ID isn;t touching the physical ram. you can run disk utility and try the verify disk to see if there are any errors. it might give you a starting point.

also, you say it's got a 1TB SSD; are you running TRIM? multiple writes and deletes on SSDs can be problematic over the long run if TRIM isn't running to clean up the free space. if you're not running TRIM, try this: reboot holding down command-s, booting into single user mode. when all the text gets finished loading, at the prompt, run fsck -fy. if you get any errors, keep running it until you don't see any. when you're done, type exit and your system will boot normally. fsck -fy "forces" a free space clean up on SSDs -- sort of a brute force TRIM -- and is a good thing to run every now and then until Apple includes native TRIM support for 3rd-party drives.

if you've got an Apple SSD in there, you might try it anyway -- it couldn't hurt.

 

[rich apollo]

The reason the resolutions are so high, is that these pieces are built to 1/4 size. Eventually some of these will output to 45 feet.

mojoprime, yes TRIM is enabled on the SSD drive(s) (they are Apple supplied drives - the System Report indicates that TRIM is enabled). The drive(s) are not full.

Postscript generation works consistently. Why wouldn't direct export to high-res PDF?