Posted on

Monitoring and the ELK Stack

Any application monitoring solution should maintain an open design, build upon proven technologies, be accessible, and require a low learning curve. The end goal is simple: provide teams with the ability to identify issues or unexpected behavior within minutes, if not seconds. The ELK Stack meets these expectations and more. In this Refcard, you'll cover the basic components of the ELK Stack, how it maps to a log analysis workflow, and step-by-step instructions for installation, configuration, and reporting.
Posted on

ELK Stack Installation and Demo

About ELK

ELK Stack is a collection of open-source products (Elasticsearch, Logstash, and Kibana) developed, managed, and maintained by Elastic until a few years ago. The ELK application is a combination of Elasticsearch (search and data analysis), Logstash (central logging, log enrichment, and parsing), and Kibana (data visualization). For example, logging user behavior on a shopping site (which browsers customers come from, which countries, which products they visit, etc.) is one of these purposes.

Elasticsearch

Posted on

Security Information and Event Management (SIEM) with Elastic

With increased Cybersecurity challenges, firms are constantly battling to bring down the Mean Time to Detect/Discover (MTTD) of security threats. This is critical for multiple aspects such as customer satisfaction, legal compliance, and creditability of the organizations. The organization needs to identify, communicate and mitigate an issue before the user does.

As an extension to my earlier article on Cybersecurity Trends, let us explore how Security Information and Event Management (SIEM) can be achieved through Elastic.

Posted on

Install and Configuration of Apache Hive-3.1.2 on Multi-Node

The Apache Hive is a data warehouse system built on top of the Apache Hadoop. Hive can be utilized for easy data summarization, ad-hoc queries, analysis of large datasets stores in various databases or file systems integrated with Hadoop. Ideally, we use Hive to apply structure (tables) on persisted a large amount of unstructured data in HDFS and subsequently query those data for analysis. 

The objective of this article is to provide step by step procedure in sequence to install and configure the latest version of Apache Hive (3.1.2) on top of the existing multi-node Hadoop cluster. In a future post, I will detail how we can use Kibana for data visualization by integrating Elastic Search with Hive. Apache Hadoop — 3.2.0 was deployed and running successfully in the cluster. Here is the list of environment and required components.

Posted on

5 Kibana Visualizations To Spice Up Your Dashboard

If you work in any way that is adjacent to data, insight, and analytics, there’s a good chance you will at least have heard of Kibana. If you haven’t then there’s no better time to be jumping on the bandwagon.

An open-source app, Kibana caters perfectly to any enterprise that needs to incorporate data discovery, navigation, and visualization. So long is the list of features and benefits that it’s impossible to cover them all in a single article. Tools like Kibana Lens showcase this beautifully. 

Posted on

Capture IoT Devices Data Via RabbitMQ

Introduction

The purpose of this article is to give you an overview of how to capture events from MQTT enabled IoT sensors/devices and monitors it via ELK stack.

After capturing events, you could either store it in event-stores or in time-series database for further processing.

Posted on

Elasticsearch Index v7.6

Elasticsearch, which is based on Lucene, is a distributed document store. It is a highly effective way of indexing your information for correlation and quick query for analysis. In this blog, I will just walk you through the steps required to create an Index, search, and visualize.

What Is an Index?

In the context of ES an index is a collection of documents.

Posted on

Sprinkle Some ELK on Your Spring Boot Logs

 

One day, I heard about the ELK stack and about its advantages, so I decided to get my hands on it. Unfortunately, I struggled to find solid documentation and supplemental content on getting started. So, I decided to write my own.

Posted on

Hunting the ELK (Stack): Data Monitoring to Visualization

Experts in the field

Made up of Elastisearch, "a search and analytics engine," Logstash, "a server-side data processing pipeline that "ingests data from multiple sources simultaneously, transforms it, and then sends it to a 'stash'," (according to Elastic's official site) and Kibana, a robust visualization tool, the ELK stack has quickly become one of the premier tools available to developers for data processing, management, and visualization. 

Whether you're just starting out with any of the three technologies, or you're a seasoned veteran, we've compiled the best that our community has to offer for basic questions about getting started to complex tutorials for real-time data management. 

Posted on

Logging Istio with ELK and Logz.io

Load balancing, traffic management, authentication and authorization, service discovery — these are just some of the interactions taking place between microservices. Collectively called a “service mesh,” these interconnections can become an operations headache when handling large‑scale, complex applications.

Istio seeks to reduce this complexity by providing engineers with an easy way to manage a service mesh. It does this by implementing a sidecar approach, running alongside each service (in Kubernetes, within each pod), and intercepting and managing network communication between the services. Istio can be used to more easily configure and manage load balancing, routing, security, and the other types of interactions making up the service mesh.

Posted on

Using the Telegraf Elasticsearch Input Plugin

The Telegraf Elasticsearch plugin is really easy to use, and it can show you a lot of useful information. To begin, let's get the InfluxDB sandbox up and running as well as an Elasticsearch node that we can populate with some data using Kibana.

Installing the InfluxDB Sandbox

For this tutorial I've opted to use the InfluxDB Sandbox, which is really easy to get running. Just clone the repository and run the ./sandbox up command and you've got all four components of the TICK stack running in their own containers. Follow the instructions here to clone the repository and get it running.

Posted on

Introduction to Elasticsearch and the ELK Stack, Part 1

SIn this article series, we are discussing Elasticsearch. In Part 1, we will start with an introduction of Elasticsearch and then will be have a brief discussion of the so-called ELK stack. In Part 2, we will then move to the architecture of Elasticsearch and what the heck nodes are, plus a look into clusters, shards, indexes, documents, replicatio,  and so on. So let's start.

Introduction to Elasticsearch

Elasticsearch is open source analytics and full-text search engine. It’s often used for enabling search functionality for different applications. For example, a blog for which you want users to be able to search for various kinds of data. That could be blog posts, products, categories, etc. You can actually build complex search functionalities with Elasticsearch, like auto-completion, handling synonyms, adjusting relevance, and so on.

Posted on

ELK Stack Overview and Use Cases

Instead of writing about what exactly ELK is, let me state the need and use cases for it.

Log Aggregation and Efficient Searching

In a very naive scenario, you have one server and lots of log messages generated by your application and system which are crucial to look at once something goes wrong. Now there are basically two problems with it:

Posted on

Server Monitoring With Logz.io and the ELK Stack

In a previous article, we explained the importance of monitoring the performance of your servers. Keeping tabs on metrics such as CPU, memory, disk usage, uptime, network traffic, and swap usage will help you gauge the general health of your environment as well as provide the context you need to troubleshoot and solve production issues.

In the past, command line tools, such as top, htop, or nstat, might have been enough, but in today’s modern IT environments, a more centralized approach for monitoring must be implemented.

Posted on

Kubernetes Logs Analysis With Elassandra, Fluent-Bit and Kibana

Elassandra simplifies your data stack by combining the power of Elasticsearch and Apache Cassandra into a single unique solution.

No Single Point Of Failure

Elasticsearch is by design a sharded master-slave architecture. The Elasticsearch master node manages the mapping changes and only the primary shards take write operations. The replica shards are read-only and can be promoted to primary shards by the master node in case of failure or maintenance. By relying on Apache Cassandra, Elassandra is master-less and has no single point of write. All nodes can process the search requests, request a mapping update, and depending on the Cassandra replication factor, take write operations.