Tips, Expertise, Articles and Advice from the Pro's for Your Website or Blog to Succeed
Valkey is an open-source alternative to Redis. It's a community-driven, Linux Foundation project created to keep the project available for use and distribution under the open-source Berkeley Software Distribution (BSD) 3-clause license after the Redis license changes.
I think the path to Valkey was well summarised in this inaugural blog post:
You can now use the Converse API in Amazon Bedrock to create conversational applications like chatbots and support assistants. It is a consistent, unified API that works with all Amazon Bedrock models that support messages. The benefit is that you have a single code base (application) and use it with different models — this makes it preferable to use the Converse API over InvokeModel (or InvokeModelWithResponseStream) APIs.
I will walk you through how to use this API with the AWS SDK for Go v2.
Knowledge Bases for Amazon Bedrock is a fully managed capability that helps you implement the entire RAG workflow from ingestion to retrieval and prompt augmentation without having to build custom integrations to data sources and manage data flows.
There are several configurations you can tweak to customize retrieval and response generation. This is done via query configuration parameters which can be applied via the console, API, or the SDK.
For the next 20 days (don’t ask me why I chose that number), I will be publishing a DynamoDB quick tip per day with code snippets. The examples use the DynamoDB packages from AWS SDK for Go V2 but should be applicable to other languages as well.
Posted: 13/Feb/2024
DynamoDB Local is a version of Amazon DynamoDB that you can run locally as a Docker container (or other forms).
It's super easy to get started:
In a previous blog, I demonstrated how to use Redis (Elasticache Serverless as an example) as a chat history backend for a Streamlit app using LangChain. It was deployed to EKS and also made use of EKS Pod Identity to manage the application Pod permissions for invoking Amazon Bedrock.
This use-case here is a similar one: a chat application. I will switch back to implementing things in Go using langchaingo (I used Python for the previous one) and continue to use Amazon Bedrock. But there are a few unique things you can explore in this blog post:
It’s one thing to build powerful machine-learning models and another thing to be able to make them useful. A big part of it is to be able to build applications to expose its features to end users. Popular examples include ChatGPT, Midjourney, etc.
Streamlit is an open-source Python library that makes it easy to build web applications for machine learning and data science. It has a set of rich APIs for visual components, including several chat elements, making it quite convenient to build conversational agents or chatbots, especially when combined with LLMs (Large Language Models).
Amazon MemoryDB for Redis has supported username/password-based authentication using Access Control Lists since the very beginning. But you can also use IAM-based authentication that allows you to associate IAM users and roles with MemoryDB users so that applications can use IAM credentials to authenticate to the MemoryDB cluster. With this authentication method, you don't need to use a (long-lived) user password. Instead, you use an authentication token generated using AWS Signature Version 4.
There are many benefits to this approach. Instead of managing username and password-based credentials, you can use IAM to centrally manage access to MemoryDB clusters. For client applications running on Amazon EC2, Amazon EKS, AWS Lambda, AWS App Runner, etc., you can inject these credentials (depending on the platform e.g. profile credentials in EC2 and instance role in App Runner) - this provides greater security.
Re: Invent 2023 has wrapped up. Before we start preparing for the 2024 edition, let me recap the announcements I was most excited about.
Here is my favorite list, in no particular order.
Conversational interaction with large language model (LLM) based solutions (for example, a chatbot) is quite common. Although production grade LLMs are trained using a huge corpus of data, their knowledge base is inherently limited to the information present in the training data, and they may not possess real-time or the most up-to-date knowledge.
Here is an example:
One of our earlier blog posts discussed the initial steps for diving into Amazon Bedrock by leveraging the AWS Go SDK. Subsequently, our second blog post expanded upon this foundation, showcasing a Serverless Go application designed for image generation with Amazon Bedrock and AWS Lambda ("Generative AI Apps With Amazon Bedrock: Getting Started for Go Developers").
Amazon Bedrock is a fully managed service that makes base models from Amazon and third-party model providers (such as Anthropic, Cohere, and more) accessible through an API. The applications demonstrated in those blog posts accessed Amazon Bedrock APIs directly, thereby avoiding any additional layers of abstraction or frameworks/libraries. This approach is particularly effective for learning and crafting straightforward solutions.
Whether it's crafting personalized content or tailoring images to user preferences, the ability to generate visual assets based on a description is quite powerful. But text-to-image conversion typically involves deploying an end-to-end machine learning solution, which is quite resource-intensive. What if this capability was an API call away, thereby making the process simpler and more accessible for developers?
This tutorial will walk you through how to use AWS CDK to deploy a Serverless image generation application implemented using AWS Lambda and Amazon Bedrock, which is a fully managed service that makes base models from Amazon and third-party model providers (such as Anthropic, Cohere, and more) accessible through an API. Developers can leverage leading foundation models through a single API while maintaining the flexibility to adopt new models in the future.
This article is an introductory guide for Go developers who want to get started building Generative AI applications using Amazon Bedrock, which is a fully managed service that makes base models from Amazon and third-party model providers accessible through an API.
We will be using the AWS Go SDK for Amazon Bedrock, and we'll cover the following topics as we go along:
In this blog post, you will learn how to build a Serverless solution for entity detection using Amazon Comprehend, AWS Lambda, and the Go programming language.
Text files uploaded to Amazon Simple Storage Service (S3) will trigger a Lambda function which will further analyze it, extract entity metadata (name, type, etc.) using the AWS Go SDK, and persist it to an Amazon DynamoDB table. You will use Go bindings for AWS CDK to implement "Infrastructure-as-code" for the entire solution and deploy it with the AWS Cloud Development Kit (CDK) CLI.
Of late, I have started to pen down summaries of my talks for those not interested in sitting through a 45-minute video or staring at slides without getting much context. Here is one for AWS DevDay Hyderabad 2023 where I spoke about "Simplifying Your Kubernetes Infrastructure With CDK8s."
CDK for Kubernetes, or CDK8s is an open-source CNCF project that helps represent Kubernetes resources and application as code (not YAML!).
The first part of this series introduced you to the AWS Lambda Go API Proxy, and how its framework/package-specific adapter implementations (for gorilla/mux, echo and net/http) allows you to run existing Go applications as AWS Lambda functions fronted by Amazon API Gateway. If you haven't already, I encourage you to take a look at it in order to get a basic understanding of the AWS Lambda Go API Proxy.
The AWS Lambda Go API Proxy also supports Gin, which is one of the most popular Go web frameworks! This follow-up blog post will demonstrate how to take an existing URL shortener service written using the Gin framework and run it as a serverless AWS Lambda function. Instead of using AWS SAM, we will change things up a bit and use the AWS CDK to deploy the solution.
The Go programming language has always had rich support for building REST APIs. This includes an excellent standard library (net/HTTP) along with many popular packages such as Gorilla mux, Gin, Negroni, Echo, Fiber, etc.
Using AWS Lambda Go runtime, you can use Go to build AWS Lambda functions. Imagine a web app that needs to authenticate users, store user data, and send emails. A Serverless approach for this would be to implement each functionality/API as a separate Lambda function. For example, you could have a Lambda function to handle user registration, another to handle user login, and so on.
Scalable data ingestion is a key aspect of a large-scale distributed search and analytics engine like OpenSearch. One of the ways to build a real-time data ingestion pipeline is to use Apache Kafka. It's an open-source event streaming platform used to handle high data volume (and velocity) and integrates with a variety of sources including relational and NoSQL databases. For example, one of the canonical use cases is the real-time synchronization of data between heterogeneous systems (source components) to ensure that OpenSearch indexes are fresh and can be used for analytics or consumed downstream applications via dashboards and visualizations.
This blog post will cover how to create a data pipeline wherein data written into Apache Kafka is ingested into OpenSearch. We will be using Amazon OpenSearch Serverless and Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless. Kafka Connect is a great fit for such requirements. It provides sink connectors for OpenSearch as well as ElasticSearch (which can be used if you opt for the ElasticSearch OSS engine with Amazon OpenSearch). Sometimes though, there are specific requirements or reasons which may warrant the use of a custom solution.