this is possibly in flow drive workflow in old hq 5 rip
you can set the dot shape for the images and vector files
drive by program called creator via mac
You're probably already aware of this, but....you'll need a different dot gain compensation curve for each of the two screen types and they will probably have different ink/water requirements on press (which may lead to a compromise in the presswork).