Sounds like the RIP is rasterizing before scaling. Everything gets rasterized before output. Try scaling the vector before sending to the RIP.
By the way, how are you sending the file to the RIP? Is it pixelated, or is this an issue with flatness (not flattening) - where smooth curves look like a collection of little straight lines?