It sounds like a server configuration issue, strange that enabling trace resolves it but maybe some setting that happens normally doesn't happen during trace.
I googled "iis maximum file download size" and there is some info about hardware limitations.
As an experiment you might try something like this:
scriptTimeout = Server.ScriptTimeout;
Server.ScriptTimeout = int.MaxValue;
response write the file here
Server.ScriptTimeout = scriptTimeout;