How to reduce the file size really depends on what you're writing to that file. Could you share an example of your workflow and/or the output file? In the meantime, here's a couple of initial thoughts:
Are maps/images involved? > select a lower resolution setting in the Image/Report Map tools.
Just data tables? > Is a formatted layout necessary? Unformatted data from the Output tool can save a lot of space.