accumulator | The Blog Pros

May 7, 2020

Accumulator and Broadcast Variables in Spark

At a high level, accumulators and broadcast variables both are Spark-shared variables. In distributed computing, understanding closure is very important. Often, it creates confusion among programmers in understanding the scope and life cycle of variables and methods while executing code in a cluster. Most of the time, you will end up getting :

org.apache.spark.SparkException: Job aborted due to stage failure:
Task not serializable: java.io.NotSerializableException: ...

July 16, 2019

Accumulator 101

Motivation

GSQL is a Turing complete Graph Database query language. Compared to other graph query languages, the biggest advantage is its support of Accumulators — global or attachable to each vertex.

In addition to providing the classic pattern match syntax, which is easy to master, GSQL supports powerful run-time vertex attributes (a.k.a local accumulators) and global state variables (a.k.a global accumulators). I have seen users learning and adopting pattern match syntax within ten minutes. However, I also witnessed the uneasiness of learning and adopting accumulators for beginners.

Packages for Store Routines in MariaDB 11.4
No categories
MariaDB 11.4 introduced many advanced features. One that grabbed my attention is the general support of packages for stored routines. Although this was previously available by activating the Oracle compatibility mode, now the feature is available gener... […]
Maintain Chat History in Generative AI Apps With Valkey
No categories
A while back I wrote up a blog post on how to use Redis as a chat history component with LangChain. Since LangChain already had Redis chat history available as a component, it was quite convenient to write a client application. But, that's not the same... […]
Knowledge Graph Enlightenment, AI, and RAG
No categories
In the previous edition of the YotG newsletter, the wave of Generative AI hype was probably at its all-time high. Today, while Generative AI is still talked about and trialed, the hype is subsiding. Skepticism is settling in, and for good reason. Repor... […]
Building an Effective Zero Trust Security Strategy for End-To-End Cyber Risk Management
No categories
You've probably heard a lot about zero-trust security lately, and for good reason. As we move more of our applications and data to the cloud, the traditional castle-and-moat approach to security just doesn't cut it anymore. This makes me come... […]
Phased Approach to Data Warehouse Modernization
No categories
A modernized database will help you focus on building innovative solutions rather than investing your time and effort in managing these legacy systems. Based on the scale of your existing data warehouse processes or jobs, it can be an enormous task to ... […]