Hardware or Process suggestions?

[Steve_S]

We are running into some issues with bloated processing times on a very large and complex VDP run we do monthly. I am not sure if we can improve this on the hardware side, like with a MacPro that is pretty built up, or if the whole process is just set up to be on the slow side. I'll try to highlight what we do:

• Mac specs - 2.6GHz Intel Core i5, 16GB RAM, 500GB SSD drive
• In Design CC2015 with SmartStream Designer, using PPML to generate imposed PDF's
• INDD file has 300+ custom image and text variable layers depending on unique client totals each month
• Database is generally 85k - 100k per run, and we batch out in 1000 increments to PDF
• Currently, processing time to generate PDF's is 48-72 hours each run if we get no timeouts or errors

Currently we are trying to find a way to dramatically cut the process time so that we can continue to grow this project. I am having no luck getting a straight answer from Adobe on whether any kind of hardware upgrade (i.e., more cores, more RAM, beefy processor) will improve the speed with which INDD handles the processing. HP has suggested Composer as an alternative ... until one of their people saw our INDD file and our logic scripts and simply said “You have how many layers and how many variables? Oh my…”

Does anyone out there run anything similar to this? And if so, any suggestions you have for improving the processing time?

No solicitations for software please.

Thanks!

 

[MailGuru]

If I understand correctly, you're using InDesign with SmartStream Designer to produce PPML's that eventually end up to be imposed PDF's. We don't use SmartStream (we use XMPie), but, will not SmartStream output directly to PDF/VT instead of PPML? We had a similar problem and sped up our processing significantly by outputting directly to PDF/VT.

 

[Steve_S]

I will check on that - thanks for the tip MailGuru! I know when we first made the switch to the Indigo, we got rid of XMPie because Designer was so much more efficient for our particular file setup. At that time, our DFE was not supporting PDF/VT output from our layout, but it could very well be that now it is a go.

 

[seratne]

I believe the export on InDesign is single threaded. So, more cores wouldn't help much. A faster processor with less cores would actually be faster. A couple things you could try is to not subset fonts (embed fully), and don't downsample or recompress images.

I don't have much experience with SmartStream. I demoed it for a bit, but adding custom logic was a slow process, and slowed down composition a lot. Simply writing a single rule to reformat a number increased processing time.

I hate to suggest it, especially if you're familiar with your current solution. But maybe try another variable data software. The InDesign based ones like SmartStream and XMPie and Designmerge, are great because you don't have to worry about the formatting changing moving to another platform. But the speed difference is pretty big.

 

[Steve_S]

Thanks! I was very curious if there was an inability to access multi-core and/or hyperthreading technology from In Design.

 

[Tim-Ellis]

Hello Steve,

Good news. You already have the solution. It's a bit old fashioned but it's paid for.

If you are printing your job on a HP Indio Press then forget PPML and move to JLYT (the native language of the Indigo). You keep all your current SmartStream files and rules, just create a Rich JLYT as an output file. This is stored on the press and then you keep a folder of any variable images on press too. You copy the .CSV data file to the press and load the whole job on Press.

Downside:

You will need to install any fonts on the press using the Font Manager in the HP Production Manager. The fonts are loaded as .PFB files. HP has a conversion tool for this but its a legacy mac tool. You might have buy FontLab Studio or download FontForge to convert the font files.

JLYT has some odd rules about font point sizes being in integers and the angle of a text box.

Upside:

Nothing loads faster on a HP press than a JLYT.

Ask your Rep / Solutions Architect about it.

Good Luck!

 

[Steve_S]

OK thanks. We'll check that out. We have a meeting with HP Friday to discuss this issue.

 

[DCurry]

I can tell you that it is the use of layer rules to turn layers on and off per record that really slows things down in SmartStream desktop version. I do have Composer and it pretty much chews through anything we throw at it. While I don't believe I've ever fed it a template with 300 layers, I have some jobs that are probably 75 layers and Composer has no problem at all. I'd be happy to test your template and report back the processing time, although it might be better to try with a slightly smaller list so I can still get my own work out! PM me if you want.

With Composer, all fonts need to be PC-compatible (TT or OTF, and it does not like the dFont flavor of TT.) You can also tell it to split the output into chunks based on record count, so you just tell it to batch in 1000s and it does the rest so you don't need to keep submitting. Overall it is much more reliable for us than the desktop version.

 

[DCurry]

Hello Steve,

You will need to install any fonts on the press using the Font Manager in the HP Production Manager. The fonts are loaded as .PFB files. HP has a conversion tool for this but its a legacy mac tool. You might have buy FontLab Studio or download FontForge to convert the font files.

JLYT has some odd rules about font point sizes being in integers and the angle of a text box.

It's been a while since I've used JLYT at a previous employer, but I thought the use of Rich mode negates these issues?

 

[Steve_S]

Thanks DCurry. We're trying to avoid that route because our design and print stream is 100% Mac. But, if we get to the point where we have no option, I may take you up on that offer. We do have a trial period that we have available to use Composer if we decide to do so. And we'll find out more about the JLYT on Friday.

 

[DCurry]

We're all Mac on the prepress side, too. I build all the templates on the Mac, then package up the template for Composer and there are no cross-compatibility issues. The font thing I mentioned is really the only thing to watch out for, and when you pack up the file for Composer it will tell you if any fonts might be problematic.

 

[Tim-Ellis]

It's been a while since I've used JLYT at a previous employer, but I thought the use of Rich mode negates these issues?

Hello Dan,

I believe you are correct about Rich Mode. We were all Fast Templates and SNAP fonts until we moved to GMC Inspire.

Am I right though that Rich Mode mean the Mac does all the heavy lifting. If you can live with the limitations of Fast Templates then you print job A while you RIP job B leaving the prepress operator to log Into Social Media Site C.