Posted on

Solving Performance Hotspots With Memory Pooling in Go

Solve performance hotspots like solving a puzzle.


You may also like: Optimizing Database Performance and Efficiency

Introduction

Igneous is an unstructured data management company. We move bytes back and forth — that’s what we do. From a low-level programming perspective, we want to touch each byte as few times as possible in order to make things go fast. In a garbage-collected language like Go, that also extends to minimizing how many bytes we allocate for each byte that is moved. One technique at hand for minimizing memory allocations is memory pooling.

Posted on

Implementing a User-Space NFS Client in Go

Introduction

Our backup application, which runs over an NFS (Network File System) mount, is designed for high performance. While developing our backup solution, we found we had to implement our own NFS client (in user space): By using our own client, we are able to fine-tune the load that the application imposes on any given server; we adapt to slow servers by backing off and to fast servers by increasing the load. Additionally, our client can list enormous directories (often with many millions of entries) piecemeal in a way that overwhelms neither client nor server.

In some ways, the NFS protocol is just another encoding. There are already many encoding packages in Go (json, xml, etc.) and like those, our team used Go’s reflection facility to get going as fast as possible. Quickly, we found reflection was the bottleneck to higher performance.