Recover From an Oops With CockroachDB

This tutorial demonstrates the ability to recover from accidental truncate or table drop.

Motivation

This scenario came out of a real-world situation where I got paged in the early hours of the day by a contractor evaluating our product on behalf of a customer. The contractor was in panic and concerned that she'd just lost four days worth of work by executing an older version of her application which included a hardcoded TRUNCATE statement. There was no backup available either. We were faced with a situation where we were going to miss customer timeline and lose the precious time reloading the data. We are talking about a multi-TB dataset. Once I had some coffee and we huddled to disucss next steps, we agreed the best next option is to leverage CockroachDB capability where dropped data does not leave the filesystem until garbage collection is complete, which by default is set to 25hours and luckily for us, that property was not changed in the cluster. Needless to say, we were able to recover the lost data following the procedure below.