Imagine a scenario where you need to change a JVM parameter, but you can’t or is not a good solution to changing the start script of your server(s).
One of the challenges we had, when we were working with containers, was a way to change a parameter to a JVM without building the docker image again. The application at the start time should read a JVM parameter where a _user profile_was defined. For specific reasons we sometimes need to change this profile, for instance, to use a more controlled user where we can debug an issue. In these situations, we want to stop the container/pod, change the profile and start again, or even start a different container with a different profile.
An in-memory database is not a new concept. However, it is associated too closely with terms like "cache" and "non-persistent". In this article, I want to challenge these ideas. In-memory solutions have much wider use cases and offer higher reliability than it would seem at a first glance.
I want to talk about the architectural principles of in-memory databases, and how to take the best of the "in-memory world"— incredible performance — without losing the benefits of disk-based relational systems. First of all, how to ensure data safety.
In our previous article from this series we introduced a use case for a cloud factory, deploying multiple private clouds based on one code base using the principles of Infrastructure as Code.
The process was laid out how we've approached the use case and how portfolio solutions are the base for researching a generic architecture.
This article launches a new series exploring a cloud factory architecture. It's focusing on presenting access to ways of mapping successful implementations for specific use cases.
It's an interesting challenge creating architectural content based on common customer adoption patterns. That's very different from most of the traditional marketing activities usually associated with generating content for the sole purpose of positioning products for solutions.
In the following, I will share my thoughts about how to set up a PostgreSQL Database in Kubernetes with some level of high availability. For that, I will introduce 3 different architectural styles. I do not make a recommendation here because, as always every solution has its pros and cons.
1. Running PostgreSQL Outside of Kubernetes
In the first scenario, you run PostgreSQL outside form Kubernetes. This means Kubernetes does not know anything about the database. This situation is often a result of a historical architecture where PostgreSQL was long before Kubernetes in an evolving architecture.
Sustainable architecture has experienced a huge rise in popularity in the past few decades, and it’s showing no signs of stopping. Many architects are trying to minimize the negative environmental impact of their buildings with sustainable and eco-friendly designs, including Singapore-based firm WOHA Architects. View this post on Instagram A post shared by WOHA Architects […]
If you zone out every time someone mentions “Kubernetes,” “containers,” or “pods,” this article is for you. No complex diagrams involved!
As a front-end developer, you don’t have to know how to configure an infrastructure from scratch. However, if you have a basic understanding of how it works, you can deploy and rollback your applications more independently while also being more informed during conversations about this topic.
Let’s start with web searching what Kubernetes is.
Kubernetes is an open-source container-orchestration system for automating computer application deployment, scaling, and management.
Simply put, a container is like a virtual computer that you can create, use, destroy and reboot remotely.
Now imagine this computer is actually in a cloud, running alongside thousands of other virtual computers. And all of this runs in a real computer which is powerful as heck. Your computer is now a container in the cloud.
But wait — how will you create and control this virtual computer? How will it communicate with other computers in the same cloud? And how will it communicate with the world? Sounds like you need a way to orchestrate all of this. Hence: our maestro, Kubernetes (or K8s for short)!
You may have also heard the term pods. What are those, and where do they fit here? Well, simply put, pods are groups of one or more containers. They are the smallest deployable unit in K8s, like an atom.
Imagine you’re building a WordPress website and you need a computer running PHP and a MySQL database. Running both in the same computer might be too heavy; so you could instead create one container for PHP, one for MySQL, and K8s will help make them communicate.
Then, you’d group these two containers in a pod, which represents the entire application. That means you can now start and kill an entire application through pods.
Now’s a good time to ask: what happens when you deploy an app in this setup?
K8s creates a new pod, redirects the traffic to it, and when it‘s sure everything‘s working, it kills the old pod. Again, more entities are involved to control and redirect requests, but we’re leaving that out today.
However, sometimes the deploy breaks something and we have to rollback our application to the previous version. Imagine everything’s on fire, and we have to start all of those computers from scratch — install Linux, Node, Git, clone the repository, install dependencies, build the app… that would take forever! If only there was a faster way, like taking a snapshot from the past to quickly restore everything to…
Enter: images! You probably have heard this term a lot too. An image is like a backup of a container with everything already installed and configured. A new image is generated with your Continuous Integration (abbr>CI) every time you push to the main branch of your repository, and it‘s then replicated into new containers when they’re created.
And what are they good for? Well, mainly two things: the first one is restoring to the previous image quickly, like our example above. But they‘re also useful when your website has a bunch of traffic and just one computer won’t be able to handle it.
When you have an image, you can create as many identical containers as you want and replicate that image across all of them, serving the exact same contents.
All done! We just covered the basics on how the infrastructure of an application works, and now hopefully you can extrapolate into whatever tools your project is using.
Many thanks to Eduardo Shiota for enabling me to explain this!
“BankNext” is on an ambitious digital transformation phase and desires to make its customer onboarding process seamless. After an elaborate functional flow analysis, BankNext implemented an orchestration architecture to collaborate between various microservices.
Business Flow
Initiation: Prospective customer initiates the joining process on BankNext
Prechecks: BankNext first invokes a Screening MSvc and a Deduplication Msvc to study the prospect and ensure that this entity is not already present in the system
Processing (after prechecks pass): CustomerMgt Msvc to create the Customer entity and AccountMgt Msvc to create the Account entity for this customer.
Compliance: Monitoring Msvc for the initial period to check suspicious activities and Recommendation Msvc to provide customer delight based on customer preferences.
Architecture Positives
BankNext is quite happy with this architecture due to the following:
This article outlines an approach to model NIST’s Zero Trust Security Architecture while migrating to MS Azure but still working with hybrid cloud deployments, and using tools and services offered by Azure.
What Is a Zero Trust Architecture (ZTA)?
The term ZTA has been in use in the domain of enterprise security models and architectures for organizations since 2010 when Forrester coined the term but became popular after NIST published it as a framework (SP 800-207, final version published in Aug. 2020). ZTA further got a lot of visibility after the US govt recently mandated all Federal agencies to adopt ZTA.
In our previous article from this series shared a look at the logical common architectural elements found in a retail data framework solution for retail organisations.
The process was laid out how we've approached the use case and how portfolio solutions are the base for researching a generic architecture.
This year the Red Hat Summit event is a bit different as we bridge the gap from pandemic reality to hopefully a form of normalcy.
As the Red Hat Summit site explains to us, this "...event is expanding to become an all-new, flexible conference series, consisting of a 2‑part immersive virtual experience as well as a global tour of small-scale, in-person* events. This series will create collective opportunities to share experiences, innovations, and insights."
Becoming data-driven is one of the many key goals that companies today are striving towards. By empowering their employees and stakeholders with data, organizations can foster insightful decision-making and create a competitive advantage. But how can we effectively achieve this and unlock the full potential of our data? Our guest for today may have the answer.
In this episode of Cocktails, we’re joined by the creator of the Data Mesh who shares with us her journey on how she came up with the concept and the challenges associated with evangelizing it. We also talk about the specifics of what this paradigm shift in data platform architecture is, explore how it’s different from existing data concepts, and dive into how organizations prepare for data mesh.
The aim of the following article is to share with you some of the most relevant guidelines in cloud-based big data-based projects that I’ve done in my recent mission. The following list is not an exhaustive one and may be completed according to each organization/project specifications.
Guidelines for Cloud-Based and Data-Based Projects
Data Storage
Use data partitioning and/or bucketing to reduce the amount of data to process.
Use Parquet format to reduce the amount of data to store.
Prefer using SNAPPY compression for frequently consumed data, and prefer using GZIP compression for data that is infrequently consumed.
Try as much as possible to store a big enough file instead of many small files (average 256MO - 1GB ) to improve performances (R/W) and reduce costs — the file system depends on the use case needs and the underlying block storage file system.
Think about a DeltaLake/IceBerg framework before managing schema evolutions and data updates using custom solutions.
“Design by query” can help improving consumption performances — for instance, you can store the same data in different designs in different depths depending on the consumption pattern.
Secure data stored on S3 using an adapted model using versioning and archiving.
Data Processing
When developing distributed applications, re-think your code in a way to avoid as much as possible data shuffling as it leads to performance leaks.
Small table broadcasting can help achieve better performances.
Once again, use Parquet format to reduce the amount of data to process thanks to PredicatePushDown and ProjectionPushDown.
When consuming data, use as much as possible data native protocols to be close to data and avoid unnecessary calls and protocols overhead.
Before choosing a computation framework, identify first if your problem needs to be solved using parallelization or using distribution.
Think to merge and compact your files to improve performances and reduce cost while reading (Delta.io can help achieve that),
Data Locality
Move the processing next to data and not the opposite — data size is generally higher than the jobs' or scripts' sizes.
Process data in the cloud and get only the most relevant and necessary data out.
Limit the inter-region transfer.
Avoid as much data travel as possible between infrastructures, proxies, and patterns.
Various
Use a data lake for analytical & exploratory use cases and use operational databases for operational ones.
To ingest data, prefer using config-based solutions like DMS/Debezuim rather than custom solutions. Also prefer also using CDC solutions for long-term running ingests.
Make structures (table, prefix, path…) that are not aimed to be shared privately.
TiDB Operator is an automatic operation system for TiDB in Kubernetes. As a Kubernetes Operator tailored for TiDB, it is widely used by TiDB users to manage their clusters throughout the entire lifecycle, and thus boasts of an active developer community.
In our 
n our 
In our 
In our 
In our