Squashed Bugs, Served Hot and Fresh With Failure Rate Heatmaps

Learn how to squash bugs!

To err is human; to introduce bugs is to be a developer, and to debug is part of everyday life. Debugging is not just about the fix — sometimes it's the drudgery of trying to locate the root cause that kills passion, especially when you have thousands or even millions of lines of code. A former Oracle database developer, who helped maintain Oracle v12.2 and its 25 million lines of C code, described his struggling life with bugs this way:

  • Spend weeks trying to understand the 20 different flags that interact in mysterious ways to cause a bug.
  • Find out that his fix causes 100~1000 failed tests.
  • Randomly pick some of the tests and dive into the code again. It seems 10 more flags are related to this bug.
  • Finally, after a few weeks or months, get the combination of flags right and succeed with zero failing tests.

He was frustrated with the situation, and this is something we want to avoid in our projects!