Apache Avro to ORC Using Apache Gobblin

Apache Avro and Apache ORC

Apache Avro and Apache ORC (Optimized Row Columnar) are top-level projects under the Apache Software Foundation. Fundamentally, they are data serialization formats with different strengths.

Apache Avro is an efficient row-based binary file format for serializing data during transfer or at rest. It uses a schema to define the data structure that has to be serialized, and the schema is collocated and stored as part of Avro’s data file. As frequently needed in big data space, Avro was designed to support data evolution by allowing the augmentation of new fields to the data structure without the need for a complete recompilation of the code that uses it.

Build an Advanced RAG App: Query Rewriting
No categories
In the last article, I established the basic architecture for a basic RAG app. In case you missed that, I recommend that you first read that article. That will set the base from which we can improve our RAG system. Also in that last article, I listed s... […]
Contexts in Go: A Comprehensive Guide
No categories
Contexts in Go provide a standard way to pass metadata and control signals between goroutines. They are mainly used to manage task execution time, data passing, and operation cancellation. This article covers different types of contexts in Go and examp... […]
Next-Gen Lie Detector: Stack Selection
No categories
The first lie detector which relied on eye movement appeared in 2014. The Converus team together with Dr. John C. Kircher, Dr. David C. Raskin, and Dr. Anne Cook launched EyeDetect — a brand-new solution to detect deception quickly and accurately. This... […]
Applying the Pareto Principle To Learn a New Programming Language
No categories
In this article, I will discuss how you can apply the Pareto principle to quickly learn a new programming language and start solving real-world problems while you develop a deeper understanding of the programming language. What Is the Pareto Principle?... […]
Enhance IaC Security With Mend Scans
No categories
Whether on the cloud or setting up your AIOps pipeline, automation has simplified the setup, configuration, and installation of your deployment. Infrastructure as Code(IaC) especially plays an important role in setting up the infrastructure. With IaC t... […]