Recently made a decision for an app I'm working on to accumulate data in batches AND THEN to insert it into the database with one statement rather than adding 1,000 records through one thousand separate inserts.
Is this done better through file or memory? If I write the segments of the bulk-insert statement to a file and then execute from the file, the security people will complain and say that I'm writing self-modifying code.
Yet, if I try to assemble the individual segments in memory, I run the risk that one of my end users will try to feed it too much data at once and that the thing will choke.
What's the lessser of two evils here? I think I might just be better off long term equipping the software to detect big batches and to break them up followed by execution from memory?
