Python – Optimization
Hey Gang!
I'm hitting a point with my (python/mysql/linux) app for processing large amounts of network records where I need to make a design decision.
I definitely want my app to have high performance. Because optimization as a skill set is so rare there is no reason not to employ it if you have it. No one can copy you because innovation is not what most tech start-ups do and they grew up coding on pay-per-flop architectures.
My methodology is to get the data into the database and to let the database do database things. I got very good advice here at Dani Web to switch to MySQL. This has been a tremendous time saver!
And yet some things I can rely upon python and python alone to do.
I am trying to minimize those things. For example, I plan to hand-write warehousing drivers in ANSI-C to get the data into the database without python.
And yet not everything can be accomplished in C.
Do any of you have any general advice about python optimization? I have tried all of the obvious things like optimizing the interpreter, things that have always worked with perl or ruby. Python has been less than cooperative.