We’ve recently had an interesting opportunity to experiment with deploying modern Python workers at Rainforest. We used to host most of our stack on Heroku[1], but it was not a good fit for this particular use case. This post explains the challenge we were facing and how we solved it, while also mentioning a bunch of cool tools that make development and deployment much less painful than it was a few years ago.
Dynamic Scaling for Python Workers
So what was our task? We wanted to run Python workers, each for some time between a few minutes and a few hours. We also wanted to have the flexibility to support thousands of workers running simultaneously during high-traffic periods without paying for the infrastructure in times of low demand – basically dynamic scaling. When you’re used to building web applications, where a request taking more than 1s is considered bad and your demand is a little more constant, you have to slightly change your perspective and probably the tools you’re using. Heroku can work great for some things, but dynamic scaling is not really one of them (things like HireFire can help, but we found that we wanted a bit more flexibility).