Open your large and small pdfs. On each, click File>Save As Other>Optimized PDF. Then, click the "Audit Apace Usage" button. This will help see where your extra space is coming from. It really depends on what effect are being preserved, color profiles, and etc.