Trino, Superset, and Ranger on Kubernetes: What, Why, How?

This article is an opinionated SRE point of view of an open-source stack to easily request, graph, audit and secure any kind of data access of multiple data sources. This post is the first part of a series of articles dedicated to MLOps topics. So, let’s start with the theory!

What Is Trino?

Trino is an open-source distributed SQL query engine that can be used to run ad hoc and batch queries against multiple types of data sources. Trino is not a database, it is an engine that aims to run fast analytical queries on big data file systems (like Hadoop, AWS S3, Google Cloud Storage, etc), but also on various sources of distributed data (like MySQL, MongoDB, Cassandra, Kafka, Druid, etc).  One of the great advantages of Trino is its ability to query different datasets and then join information to facilitate access to data. 

How to Develop Your Distributed SQL Statement in Apache ShardingSphere

In the previous articles “An Introduction to DistSQL” and “Integrating SCTL Into DistSQL’s RAL— Making Apache ShardingSphere Perfect for Database Management”, the Apache ShardingSphere committers shared the motivations behind the development of DistSQL, explained its syntax system, and impressively showcased how you can use just one SQL to create a sharding table.

Today, to help you gain a better understanding of DistSQL and develop your own DistSQL syntax, our community author analyzes the design & development process of DistSQL and showcases how you can implement a brand new DistSQL grammar in four stages of the development life cycle (i.e. demand analysis, design, development & testing).

Developing Event-Driven Microservices

This is the second in a series of blogs on data-driven microservices design mechanisms and transaction patterns with the Oracle converged database. The first blog illustrated how to connect to an Oracle database in Java, JavaScript, Python, .NET, and Go as succinctly as possible. The goal of this second blog is to use that connection to receive and send messages with Oracle AQ (Advanced Queueing) queues and topics and conduct an update and read from the database using all of these same languages.

Advanced Queuing (AQ) is a messaging system that is part of every Oracle database edition and was first released in 2002. AQ sharded queues introduced partitioning in release 12c and is now called Transaction Event Queues (TEQ).

Integrating SCTL Into DistSQL’s RAL

In the previous article “An Introduction to DistSQL” written by Haoran Meng, the Apache ShardingSphere Committer shared the motivating reasons behind the design of DistSQL, explained its syntax system, and impressively showcased how you can use one SQL to create a sharding table.

Recently, the ShardingSphere community has redesigned the SCTL grammar and the execution engine, integrating SCTL into the DistSQL syntax system. Now RAL contains the old SCTL function, making ShardingSphere’s command language even more convenient for database management. Today, our community author would like to introduce the changes and elaborate on how you can use the new RAL command lines. We always pursue a better user experience, and the upgrade we developed this time is just another typical example.

AWS CloudWatch + yCrash = Monitoring + RCA

AWS Cloud Watch + yCrash = Monitoring + RCAWe had an outage in our online application GCeasy on Monday morning (PST) Oct 11, 2021. When customers uploaded their Garbage Collection logs for analysis, the application was returning an HTTP 504 error. HTTP 504 status code indicates that transactions are timing out. In this post, we would like to document our journey to identify the root cause of the problem.

Application Stack

 Here are the primary components of the technology stack of the application:

Container Creates Instant Database API

In this tutorial, we’ll show how to use ApiLogicServer to create, customize and run a Database based API. API Logic Server is an open-source Docker container. With the commands shown below, you get:

  • Working Software, Now:
    • A database API server, to unblock UI development.
    • A multi-page web app, to engage Business Users — early in the project.
    • Declarative logic using unique spreadsheet-like rules — 40X more concise than code, extensible with Python — for remarkable business agility.
  • Customizable projects, using a standard language and tools. Operate in a cleanly isolated, containerized environment that matches your deployment architecture.

TL;DR — Create Database API and Basic Web App

Create the sample project in a minute or two, as follows. With Docker started, enter these Terminal commands (Windows, use Powershell):

Set Up a Database Diagram Using a Stored Procedure In SQL Server

Steps to be Followed:

  1. Create tables
  2. Create stored procedure using inner join between two tables
  3. Find out stored procedure syntax as text using SQL query
  4. Execute stored procedure to get results
  5. List of tables used in a stored procedure
  6. Then create a database diagram

Step 1

Create a table named: Supplier_Type_Master.

SQL syntax

Fetch T-SQL With Passed Parameter Value Using Stored Procedure

Introduction

Using "sys.objects," we can get a list of tables and stored Procedures created in the Database. SQL Server now supports table-valued parameters, which allow us to send data tables as parameters to Stored Procedures. It still uses the same ADO.NET API. Using SYSOBJECTS and SYSDEPENDS, we can get all the tables, stored procedures, and other database object-related information.

Description

You can apply the OBJECT_ID, OBJECT_NAME, and OBJECTPROPERTY() built-in functions to the objects shown in sys.objects. It contains a row for each user-defined, schema-scoped object that is created within a database. sys.objects does not show DDL triggers because they are not schema-scoped. Table Valued Parameter allows a table (i.e multiple rows of data) to be passed as a parameter to a stored procedure from T-SQL code or from an application. It was not possible to pass a table variable as a parameter to a stored procedure in old versions of SQL Server.

What Is a Smart Database Proxy?

Smart database proxies may not be familiar to many people, and it's a shame because they can solve many difficult problems elegantly. This article explains what they are, what they do, and when they are useful.

A Quick Comparison

Let me start with a seemingly unrelated topic: web pages. Bear with me, it will all come together in a minute.

Database Drivers: Chauffeuring Your Data to Where it Needs to Go

Most, if not all, companies deal with complications and integration headaches somewhere in their data pipeline due to an inability or difficulty of connecting certain systems. Sometimes you have to add yet another technology to the lineup just to connect different systems and get your data to where it needs to go. However, in this modern-day, less is more. Most technologies that emerge are all about being more efficient and providing more functionality in a smaller package. If you can meet your data management needs with fewer tools, then it’s a win-win for cost-effectiveness, efficiency, and ease of use. Enter database drivers.

The Magical Adaptors

To put it plainly, each computer system needs some sort of adaptor or tool to be able to connect to other computer systems that are not the same. You can think about this in the physical or interior (computer system) sense. For a physical example, we all know the major frustration that comes along with upgrading your phone, when your chargers and headphones no longer fit in the ever-shrinking port on the bottom of the phone. However, you can buy an adapter to serve as a “middle man” to enable the connection. With computer systems, it’s the same but different. A database driver works like that physical phone adaptor, but instead of having to invest in an additional product to add to the tech stack, you can develop an adaptor/connector to extend the database functionality. Like an extension to a software package.

Agile Design Automation With Logic Bank

Automation

As engineers, we are always mindful of the need to minimize cost and time.  Experience has taught us that while automation is valuable in reducing code, we must also preserve design flexibility and enable agile iterations.  Let's break that down a bit, and then illustrate with an example.

Automation (Reduce Routine Code — Executable Specifications)

To warrant the learning curve, automation must significantly reduce routine coding.  25%  reduction in code is not compelling enough.

Introduction to PDO Extension: How to Use PDO for Different Databases

In this article, we will talk about what a PDO is, why is it necessary, and how to work with it.
Let's start with the fact that for a long time php-programmers have been discussing the fate of the native functions to work with the database: mysql_connect, mysql_query, and others. The fact is that in upcoming versions of PHP data will be erased, these functions will go "Obsolete". Thus PHP developers should force on other tools to work with the database.

Currently, there are two alternatives solutions; mysqli extension and expansion of PDO. In this article, we will discuss PDO.
PDO - PHP Data Objects - is a layer that offers a versatile way to work with multiple databases.
This layer has several advantages:
  • PDO allows you to work with different databases, such as: MYSQL, Oracle, PostgreSQL and many others.
  • PDO allows you to work with a prepared statement
Now let's talk about all this in more detail.

Connecting to a Database Using PDO

Before you start working with PDO, you must make sure that you have the correct extension - php_pdo_mysql.dll. This can be verified through the function phpinfo(). If everything is okay, you are good to go.

First we need to connect to the database:
PHP
 




xxxxxxxxxx
1
15


 
1
/ / MuSQL
2

          
3
    $Database = new PDO ("mysql: host = $ host; dbname = $Databasename", $username, $password); 
4

          
5
/ / PostgreSQL
6

          
7
    $Database = new PDO ("pgsql: host = $ host; dbname = $Databasename", $username, $password);
8

          
9
/ / MS SQL
10

          
11
    $Database = new PDO ("mssql: host = $ host; dbname = $Databasename", $username, $password); 
12

          
13
/ / SQLite
14

          
15
    $Database = new PDO ("sqlite: my / database / path / database.db");



As you can see from the example, each database connection string is slightly different from each other
You should always use try/catch when you are trying to connect to the unit, so that on exception we do not disclosed any data to the user.
PHP
 




xxxxxxxxxx
1


 
1
try {
2
    $Database = new PDO ("mysql: host = $ host; dbname = $Databasename", $username, $password); 
3
} 
4
catch (PDOException $ e) { 
5
    echo "Unable to connect to database"; 
6
}



Instant DB Web Apps

Here's how to create a basic web app for your database in 10 minutes.  Let's explore the created app, and then the creation process.

Created Web App

As shown below, the apps are

Should You Use DynamoDB? (Part 1)

Selecting a proper technology for a new project is always a stressful event. We need to find something that will fit all existing requirements, does not restrict further growth, allows us to achieve necessary performance, does not put a heavy operational burden, etc. It’s only natural that selecting a database can be tricky.

In this article, I would like to describe the DynamoDB database created by AWS. My goal is to give you enough knowledge so you would be able to answer a simple question: “Should I use DynamoDB in my next project?” I will describe in what cases DynamoDB can be used efficiently and what pitfalls to avoid. I hope this will help you to make your life easier.