Python – MySQL – Too many connections?

Featured Imgs 23

A script that was working fine on 100 records per batch choked when I fed it 1,000 records.
It gave:

mysql.connector.errors.OperationalError: 1040 (08004): Too many connections

So I went ahead and made sure to connect.close() everywhere that I opened a connection, and the script made it much further but eventually crashed, giving the same error.
Obviously, increasing the connection limit would only encourage sloppy coding? It would be better if I developed a methodology to prevent this type of crash. Because if this were PHP it wouldn't be happening, I admit that.

How can I prevent this type of error from happening again, short of switching to PHP?

Database Design – Timestamps

Featured Imgs 23

This week I'm working on the data warehousing capabilities of my application. Now most people here who have built more than a couple of applications can tell you how useful it is to include a datestamp on records or requests as they come in. The advantages are numerous.

So much so that I'm now fighting the temptation to include a datestamp on every table. Won't that bloat my database?

I suppose what I'm looking for is guidance on making a per table decision to include or exclude the datestamp. Because if space were free I'd put one on every darn table! But space isn't free and my app is likely to generate a factor of five to ten times the amount of metadata per data and so datestamps would put me in danger of doubling that again, or trebling it. That's fine up to a point but will impact load times.

Python Undefined Variable – Mysql Connector

Featured Imgs 23

Hello Gang! I'm grateful once again for the advice to use MySQL over Postgres. It's been a pretty smooth experience so far. Unfortunately I had to migrate to a different host and have been getting an undefined variable error.

Mysql is Running. I used the package manager to get python-mysql.connector installed. And yet I keep getting an undefined variable error.

Here is the output:

Exception creating initial db connection
Error:  module 'ssl' has no attribute 'wrap_socket'
Error type:  <class 'AttributeError'>

Here is some test code if you need to replicate it:

import base64
import mysql.connector as mysql
try:
    connection=mysql.connect( host="localhost", database="iheartdaniweb", user="funuser", password="supersekritpass") 
except Exception as error:
    print("Exception creating initial db connection")
    print("Error: ", error)
    print("Error type: ", type(error))
connection.autocommit=True

Thanks in advance gang! Unfortunately the interwebs were no help with this one or I wouldn't bother you guys! Thanks again!! !!

Bulk Data Inserts – From File or Memory

Featured Imgs 23

Recently made a decision for an app I'm working on to accumulate data in batches AND THEN to insert it into the database with one statement rather than adding 1,000 records through one thousand separate inserts.

Is this done better through file or memory? If I write the segments of the bulk-insert statement to a file and then execute from the file, the security people will complain and say that I'm writing self-modifying code.

Yet, if I try to assemble the individual segments in memory, I run the risk that one of my end users will try to feed it too much data at once and that the thing will choke.

What's the lessser of two evils here? I think I might just be better off long term equipping the software to detect big batches and to break them up followed by execution from memory?

Python – Optimization

Featured Imgs 23

Hey Gang!

I'm hitting a point with my (python/mysql/linux) app for processing large amounts of network records where I need to make a design decision.
I definitely want my app to have high performance. Because optimization as a skill set is so rare there is no reason not to employ it if you have it. No one can copy you because innovation is not what most tech start-ups do and they grew up coding on pay-per-flop architectures.
My methodology is to get the data into the database and to let the database do database things. I got very good advice here at Dani Web to switch to MySQL. This has been a tremendous time saver!
And yet some things I can rely upon python and python alone to do.
I am trying to minimize those things. For example, I plan to hand-write warehousing drivers in ANSI-C to get the data into the database without python.
And yet not everything can be accomplished in C.
Do any of you have any general advice about python optimization? I have tried all of the obvious things like optimizing the interpreter, things that have always worked with perl or ruby. Python has been less than cooperative.

create a database-specific table in postgres?

Category Image 101

How do I create a table in postgres that's associated with a specific database?

Obviously from the command line one can do

Create table tablename ();

But how do I make sure that the table is associated with the correct database?

CREATE table dbname.tablename();

is giving me the error:

ERROR:  schema "dbname" does not exist