Using Redis on Cloud? Here Are Ten Things You Should Know

It's hard to operate stateful distributed systems at scale and Redis is no exception. Managed databases make life easier by taking on much of the heavy lifting. But you still need a sound architecture and apply best practices both on the server (Redis) as well as the client (application).

This blog covers a range of Redis-related best practices, tips and tricks including cluster scalability, client-side configuration, integration, metrics etc. Although I will be citing Amazon MemoryDB and ElastiCache for Redis from time to time, most (if not all) will be applicable to Redis clusters in general.

Data Management Patterns for Microservices

This is an article from DZone's 2022 Database Systems Trend Report.

For more:


Read the Report

One of the key components of microservices is how to manage and access data. The means to do that are different compared to traditional monolithic or three-tier applications. Some patterns are quite common, but others are specific and need to be evaluated before being incorporated into a solution. We will briefly go over some of these common database patterns for microservices before exploring CQRS (including how it differs from CRUD) and, finally, look at how it can be combined with event sourcing.

Manage Redis on AWS From Kubernetes

In this blog post, you will learn how to use ACK with Amazon EKS for creating a Redis cluster on AWS (with Amazon MemoryDB).

AWS Controllers for Kubernetes (also known as ACK) leverage Kubernetes Custom Resource and Custom Resource Definitions and allow you to manage and use AWS services directly from Kubernetes without needing to define resources outside of the cluster. It supports many AWS services including S3, DynamoDB, MemoryDB etc.

Use AWS App Runner, DynamoDB, and Cdk To Deploy and Run a Cloud-native Go App

Earlier, I covered a Serverless URL shortener application on AWS using DynamoDB, AWS Lambda and API Gateway.

In this blog post, we will deploy that as a REST API on AWS App Runner and continue to use DynamoDB as the database. AWS App Runner is a compute service that makes it easy to deploy applications from a container image (or source code), manage their scalability, deployment pipelines, and more.

Write Your Kubernetes Infrastructure as Go Code – Extend cdk8s With Custom Constructs

Constructs are the fundamental building block of cdk8s (Cloud Development Kit for Kubernetes) - an open-source framework (part of CNCF) with which you can define your Kubernetes applications using regular programming languages (instead of yaml). In Getting started with cdk8s, you saw how to use the core cdk8s library.

You can also use the cdk8s-plus library (also covered this in a previous blog) to reduce the amount of boilerplate code you need to write. With cdk8s-plus, creating a Kubernetes Deployment, specifying its container (and other properties) and exposing it via a Service is three function calls away?

Build a Serverless URL Shortener With Go

This blog post covers how to build a Serverless URL shortener application using Go. It leverages AWS Lambda for business logic, DynamoDB for persistence, and API Gateway to provide the HTTP endpoints to access and use the application. The sample application presented in this blog is a trimmed-down version of bit.ly or other solutions you may have used or encountered.

It's structured as follows:

Write Your Kubernetes Infrastructure as Go Code — Using Custom Resource Definitions With Cdk8s

cdk8s (Cloud Development Kit for Kubernetes) is an open-source framework (part of CNCF) using which you can define your Kubernetes applications with regular programming languages (instead of yaml). Some of the previous blogs on this topic covered the getting started experience and using cdk8s-plus library to further improve upon the core cdk8s library features. We are going to continue and push cdk8s even further.
This blog post will demonstrate how you can use Kubernetes Custom Resource Definitions with cdk8s. We will start off with a simple Nginx example and then you will use the combination of Strimzi project CRDs along with Go  cdk8s to define and deploy a Kafka cluster on Kubernetes!

I am assuming that you've have some knowledge of Kubernetes Custom Resource Definitions and have probably even used a few in the form of Operators. If not, that's ok! The Kubernetes documentation covers it quite well. You can always refer to it, come back here and follow along!

Learn How To Use DynamoDB Streams With AWS Lambda and Go

This blog post will help you get quickly started with DynamoDB Streams and AWS Lambda using Go. It will cover how to deploy the entire solution using AWS CDK.

The use case presented here is pretty simple. There are a couple of DynamoDB tables and the goal is to capture the data in one of those tables (also referred to as the source table) and replicate them to another table (also referred to as the target table) so that it can serve different queries. To demonstrate an end-to-end flow, there is also an Amazon API Gateway that front ends a Lambda function which persists data in the source DynamoDB table. Changes in this table will trigger another Lambda function (thanks to DynamoDB Streams) which will finally replicate the data into the target table.

Write Your Kubernetes Infrastructure as Go Code — Cdk8s-Plus in Action!

One of my previous blog posts covered how to get started with cdk8s (Cloud Development Kit for Kubernetes), which is an open-source framework (part of CNCF) using which you can define your Kubernetes applications using regular programming languages (instead of yaml).

You were able to set up a simple nginx Deployment and accessed it via a Service - all this was done using Go, which was then converted to yaml (using cdk8s synth) and submitted to the cluster using kubectl. This was a good start. However, since the core cdk8s library is pretty low-level (for a good reason!) the code involved a lot of boilerplate (you can refer to the code here).

Write Your Kubernetes Infrastructure as Go Code – Getting Started With Cdk8s

Infrastructure as Code (IaC) is a well established paradigm and refers to the standard practice of treating infrastructure (network, disk, storage, databases, message queues etc.) in the same way as application code and applying general software engineering practices including source control versioning, testing and more. For example, Terraform and AWS CloudFormation are widely-adopted technologies that use configuration files/templates to represent the infrastructure components.

Infrastructure-IS-Code - A Different Way of Thinking About This

Imagine you have an application that comprises a Serverless function fronted by an API Gateway along with a NoSQL database as the backend. Instead of defining it in a static way (using JSON, YAML, etc.), one can represent these components using standard programming language constructs such as classes, methods, etc. Here is a pseudo-code example:

MySQL to DynamoDB: Build a Streaming Data Pipeline on AWS Using Kafka

This is the second part of the blog series which provides a step-by-step walkthrough of data pipelines with Kafka and Kafka Connect. I will be using AWS for demonstration purposes, but the concepts apply to any equivalent options (e.g. running these locally in Docker).

This part will show Change Data Capture in action that lets you track row-level changes in database tables in response to create, update and delete operations. For example, in MySQL, these change data events are exposed via the MySQL binary log (binlog).

Build a Data Pipeline on AWS With Kafka, Kafka Connect, and DynamoDB

There are many ways to stitch data pipelines: open source components, managed services, ETL tools, etc. In the Kafka world, Kafka Connect is the tool of choice for "streaming data between Apache Kafka and other systems." It has an extensive set of pre-built source and sink connectors as well as a common framework for Kafka connectors, which standardizes integration of other data systems with Kafka and makes it simpler to develop your own connectors, should there be a need to do so.

This is a two-part blog series that provides a step-by-step walkthrough of data pipelines with Kafka and Kafka Connect. I will be using AWS for demonstration purposes, but the concepts apply to any equivalent options (e.g., running these locally using Docker). Here are some of the key AWS services I will be using:

Build a Twitter Leaderboard App With Redis and AWS Lambda (Part 2)

This is the second blog post of a two-part series that uses a practical application to demonstrate how to integrate Redis with AWS Lambda. The first post was about the solution overview and deployment. Hopefully, you were able to try it out end to end. As promised, part two will cover the infrastructure aspects (IaC to be specific) which are comprised of three (CDK) stacks (in the context of a single CDK App).

I will provide a walk-through of the CDK code which is written in Go, thanks to the CDK Go support. AWS Cloud Development Kit (CDK) is all about IaC (Infrastructure-as-code).

Getting Started With Redis on AWS the Easy Way

Like most of the services, Amazon MemoryDB for Redis is fully integrated with Amazon VPC and always launches your cluster in a VPC. It means that you cannot access it from outside your VPC. While initially exploring these services (MSK, Elasticache for Redis, etc.), I usually followed the documentation that involved setting up EC2, SSH-ing into the instance, installing/copying stuff (language, runtime, code, client, etc.), and then tried things out.

Most often, the first step is the hardest, and it's important for developers to have the least amount of friction as possible to "get going." I looked for simpler ways and found AWS Cloud9 to be super useful. It was quick, predictable, and had a bunch of useful tooling readily available.

Package and Deploy a Lambda Function as a Docker Container With AWS CDK

One of my previous blog posts covered how to build a Serverless backend for Slack using by using Lambda Function URL as a webhook. Since I wanted to focus on the application itself, the infrastructure setup part was simplified - using AWS CLI, the function was packaged as a zip file, configured and finally, a Function URL was created along with the required permissions.

In this blog, you will end up deploying the same solution, but this time using IaaC (Infrastructure-as-code) with AWS Cloud Development Kit (CDK) which is a framework for defining cloud infrastructure in code and provisioning it through AWS CloudFormation. You can choose from a list of supported programming languages (at the time of writing - TypeScript, JavaScript, Python, Java, C#/.Net, and Go (in developer preview)) to define your infrastructure components as code, just like you would with any other application!

Use Lambda Function URL To Write a Serverless App Backed by DynamoDB

Lambda Function URL is a relatively new feature (at the time of writing this blog) that provides a dedicated HTTP(S) endpoint for your Lambda function. It is really useful when all you need is a single endpoint for your function (e.g. to serve as a webhook) and doesn't want to set up and configure an API Gateway. Looks like I can't seem to get enough of it! I have written a couple of blog posts on this topic that includes a practical example of using it to build a serverless backend and then deploying that solution using AWS CDK.

This is yet another blog post (it's a short one!) that demonstrates how you can use Lambda Function URL to write a simple application backed by DynamoDB. You will be able to invoke an API endpoint exposed by the Lambda Function URL, which in turn will execute operations (GetItem, PutItem, Scan) on DynamoDB. The function is written in Go using the DynamoDB package in AWS Go SDK and AWS Serverless Application Model (SAM) is used to quickly build and deploy the solution.

Using AWS Lambda Function URL To Build a Serverless Backend for Slack

A combination of AWS Lambda and Amazon API Gateway is a widely-used architecture for serverless microservices and API-based solutions. They enable developers to focus on their applications, instead of spending time provisioning and managing servers.

API Gateway is a feature-rich offering that includes support for different API types (HTTP, REST, WebSocket), multiple authentication schemes, API versioning, canary deployments, and much more! However, if your requirements are simpler and all you need is an HTTP(S) endpoint for your Lambda function (for example, to serve as a webhook), you can use Lambda Function URLs! When you create a function URL, Lambda automatically generates a unique HTTP(S) endpoint that is dedicated to your Lambda function.