Posted on

Experiencing the Aftermath Will Make You Tougher, Wiser, and Ready For Anything

I was in charge of managing a dedicated server running Debian 7. The server hosts multiple websites with email services, the server also hosts multiple instances of a critical web application for a client who is running a business across different regions.

That day was a very important day, as the client was expecting his own client to turn up. The client wanted to make a demo of the application and show how they manage some business processes.
During that event, I got a phone call claiming client users not able to access the web application. I took the request as usual and started checking the filed issue, and a few seconds later, I got another call about other users not able to access their mailbox. It is then I realized that a very nasty thing is happening and I'm in serious trouble.  

I quickly figured out that I had made the worst mistake ever!

That day I was performing usual maintenance tasks on the server, freeing some disk space here and there. However, at some moment, I deleted critical files that belong to different services like Postgres, Mysql, mail server, etc. I didn't notice anything until I started receiving reported issues from the clients. 

It was catastrophic in all corners.
We lost three months of data as backups resided in a single place where I launched the deletion operation, there were no other copies of backups. Many services were surviving with what was left in RAM (I guess) and any respawned process was lethal for the corresponding service.
In the field, the client was badly embarrassed in front of his client, as he was cut off at the beginning of the demo. 

Users started processing customer data manually using a pen and paper. The recovering of the server was a "mission: impossible". We needed to reinstall everything from the scratch, but it was decided that the fastest recovery would be to migrate to another server with the latest Debian version.

So, I installed all the required services on the new server and restored the most recent backup. The business application was finally live.
One must learn a lot of lessons because of this disaster.  

Lessons learned:

Posted on

Making Safe Choices When Opting for Open Source in Your Business

Despite the popularity of OS software, one developer argues why open source could not be safe for your business.

Whether you are looking for a document management system or a development framework for your next business application, you might be considering open-source solutions. However, making the right choice that meets your requirements doesn't necessarily mean it's a safe choice.

Imagine a company is using a system (could be an ERP, CRM, DMS, ESB — the list goes on) for a couple of years, and that system relies on an open-source framework for that one day, for whatever reason, stops receiving maintenance updates, and the next version will be released under a different open-source license with more restrictions.