Posted on

From Spreadsheet to Database With ddlgenerator

I've been using some fun open data sets recently but I'm more familiar with databases than with traditional data science formats (this one is CSV). Importing CSV into Postgres is fairly well-established, but you need a table structure to import into ... and I was looking at a CSV with 50 columns (for my post about exoplanets on the Aiven blog). Enter this neat utility: ddlgenerator.

I used it for a CSV file and put the data into PostgreSQL but on closer inspection, I realised that this tool can ingest a great selection of different formats, including JSON, YAML, and even HTML. And it can create the table structure for any of the dialects that SQLAlchemy can speak — I expected PostgreSQL and MySQL support but the selection is pretty extensive, including SQLite, SQL Server, and even Oracle databases.

Posted on

Handy Beanstalkd Admin Console

I've been building apps with queues lately and mostly using beanstalkd as my queue because it is very simple, very fast, and, on my platform, it is [apt install beanstalkd] easy to install. I have also been using a handy web interface for beanstalkd which I like so much that I felt I ought to share! It's beanstalk-console, which is a PHP-based web interface to one (or many) beanstalkd servers.

Beanstalkd is a very simple queueing system and I've enjoyed using it from a few different tech stacks. It uses "tubes" rather than channels, supports priorities, and uses a simple string for the body of the "job" on the queue.