Day: June 17, 2024

Job scheduling is an important part of data management as it enables regular data updates and cleanups. In a data platform, it is often undertaken by workflow orchestration tools like Apache Airflow and Apache Dolphinscheduler. However, adding another component to the data architecture also means investing extra resources for management and maintenance. That's why Apache Doris 2.1.0 introduces a built-in Job Scheduler. It is strategically more tailored to Apache Doris and brings higher scheduling flexibility and architectural simplicity.

The Doris Job Scheduler triggers the pre-defined operations at specific time points or intervals, thus allowing for efficient and reliable task automation. Its key capabilities include:

In recent years, the landscape of application development has undergone a significant transformation. The rise of low-code platforms is revolutionizing how applications are built, deployed, and maintained.  

These platforms are designed to simplify and democratize the development process, enabling individuals with little to no programming experience to create functional applications.  

In the evolving landscape of front-end development, technologies like HTMX are redefining the way developers approach building modern web applications. HTMX, ranked second in the prestigious 2023 JavaScript Rising Stars “Front-end Frameworks” category,  just behind the ubiquitous React, and earning a spot in the GitHub Accelerator program. HTMX’s popularity continues with over 20k stars on GitHub, appealing to developers seeking lightweight and efficient solutions for modern web development challenges.

In this article, we explore key features of HTMX, its advantages, and use cases, while also drawing comparisons with React. By examining how HTMX differs from React and understanding the unique strengths and weaknesses of each, developers can make decisions when selecting the appropriate toolset for their projects. Whether prioritizing simplicity, rapid prototyping, or robust component-based architectures, HTMX and React offer distinct approaches.

MySQL is one of the most widely used and most popular databases for web applications. Hence, data engineers often have to create pipelines to replicate data stored in MySQL into data warehouses or data lakes to make it available for analytics.

Airbyte is an open-source data integration platform that allows you to easily build ELT pipelines from MySQL into any of the several supported destinations using log-based Change Data Capture (CDC).

Julia Evans has released what she’s saying is one of her most popular zines to date: How Git Works.

I don’t think you’d regret reading it. I imagine most of us get by with knowing just enough Git to do our jobs, but are probably using 5% of what it can really do. Being very strong with Git will almost surely benefit you in your career. Imagine helping a superior out of a sticky situation where it might look like code was lost or otherwise screwed up. Being the solution during an emotional time is clutch. Surely this pairs nicely with Oh Shit, Git!, a real classic from Katie-Sylor Miller which I see has been revitalized with Julia here.

Just the other day here at CodePen Headquarters, I saw a co-worker solve an issue with git bisect. Have you even heard of that?! Imagine there is a bug in your code, but you have absolutely no idea when it happened or where in the code it might be. That’s not a good feeling, but it’s exactly where git bisect comes in. As best I understand it, it sets the HEAD of your repo back in time some amount, and there, you test if the bug is present and you can say git bisect good or git bisect bad. Then it moves the HEAD and you keep testing and eventually it gets closer and closer to the exact commit (or at least a range of commits) where the bug happened. Then you can look at the changed files in those commits and figure out where in the code the bug may have came from. So cool!

I certainly know developers who know Git and work with it exclusively at the command line entirely as-provided. But I find it more common among the command line types that they at least have some aliases set up for the most common things they do. Those might be their own aliases, like they’ll make gco do a git checkout, but it’s worth knowing git itself allows you to make aliases within itself, which could be good since they won’t conflict with anything else. (Have I told you how long I had cp aliased to move to our local CodePen project directory? 🤣).

A much more elaborate take on git aliases is called Gut. With it, you don’t git commit anything (with all the params and whatnot you have to also pass), you gut save which launches a little wizard that asks you questions, and then it does the proper git stuff with the information you give it.

I could see that being great for a beginner, but maybe feel a little too slow as you get more comfortable at the command line. Except when it comes to the more advanced stuff and how it looks designed to get you out of binds. The fix and undo commands like awfully helpful and are the kind of things where I can never remember the proper commands.

Paweł Grzybek lays out a classic situation:

Let’s say that we are halfway through the feature, intensely focused on a task, when a critical bug needs to be fixed out of the blue. Happens to us all the time! Should we stash the current changes? Should we quickly smash git add . && git commit -m "wip" and promise that we will sort this mess out later?

His answer is no, it’s using git worktrees. It solves the issue by literally making another copy of your project on disk. And you can open and work on it separately but it all goes to the same repo ultimately. So you can leave your half-done uncommitted work on another worktree while you hop to the other to do work. Me, I’m mostly cool with git stash to tuck stuff away while I go work on something else, or even just the ol “work in progress” or “saving work” commit like Paweł mentioned. It’s not pretty but culturally it’s fine on our project. But I can see how you could get into a groove with worktrees, particularly if your editor supports it nicely.

Phew! We probably talked about Git too much, eh? I know nobody cares. Now let’s go back to just doing the 3-4 commands we know everyday, just with a few more resources in our pocket when we need them.

I gotta leave you with something else. (Digs through bag of hot links.) Ah here we go. This video rules: Flash is dead so I rebuilt it with javascript. Andrew Jakubowicz walks us through building an interface with a pretty modern and lightweight set of tools. Andrew works for Google on Lit, so it’s sort of a big excuse to show off working with Web Components, but it’s a fun ride. At 8 minutes more happens than a typical hour long video.

1. FORECAST: SNOWFLAKE.ML.FORECAST function enables businesses to forecast the metrics based on historical performance. You can use these functions to forecast future demand, Pipeline gen, sales, and revenue over a period.
2. ANOMALY_DETECTION: SNOWFLAKE.ML.ANOMALY_DETECTION function helps flag outliers based on both unsupervised and supervised learning models. These functions can be used to identify the spikes in your key performance indicators and track the abnormal trends. 
3.  TOP_INSIGHTS: SNOWFLAKE.ML.TOP_INSIGHTS function enables the analysts to root cause the significant contributors to a particular metric of interest. This can help you track the drivers like demand channels driving your sales, and agents dragging your customer satisfaction down.

In this article, I will focus on exploring the FORECAST function to implement the time series forecast model to estimate the sales for a superstore based on the historical sales.

Poor data quality can cause inaccurate analysis and decision-making in information-driven systems. Algorithms for Machine learning (ML) classification have emerged as efficient tools for addressing a wide range of data quality issues by automatically finding and correcting anomalies in datasets. There are various methods and strategies used to apply ML classifiers to tasks such as data purification, outlier identification, missing value imputation, and record linkage. The evaluation criteria and performance analysis methodologies used to measure the efficacy of machine learning models in resolving data quality issues are evolving. 

Overview of Machine Learning Classification Techniques 

Machine learning classification techniques are critical for recognizing patterns and making projections from input data. Four popular methods are Naive Bayes, Support Vector Machines (SVM), Random Forest, and Neural Networks. Each strategy has unique advantages and disadvantages. 

Stream processing has existed for decades. However, it really kicks off in the 2020s thanks to the adoption of open-source frameworks like Apache Kafka and Flink. Fully managed cloud services make it easy to configure and deploy stream processing in a cloud-native way; even without the need to write any code. This blog post explores the past, present, and future of stream processing. The discussion includes various technologies and cloud services, low code/ no code trade-offs, outlooks into the support of machine learning and GenAI, streaming databases, and the integration between data streaming and data lakes with Apache Iceberg.

In December 2023, the research company proved that data streaming is a new software category and not just yet another integration or data platform. Forrester published “The Forrester Wave™: Streaming Data Platforms, Q4 2023“. Get free access to the report here. The leaders are Microsoft, Google, and Confluent, followed by Oracle, Amazon, Cloudera, and a few others. A great time to review the past, present, and future of stream processing as a key component in a data streaming architecture.

React 19 Beta is finally here, after a two-year hiatus. The React team has published an article about the latest version. Among the standout features is the introduction of a new compiler, aimed at performance optimization and simplifying developers’ workflows.

Furthermore, the update brings significant improvements to handling state updates triggered by responses, with the introduction of actions and new handling from state hooks. Additionally, the introduction of the use() hook simplifies asynchronous operations even further, allowing developers to manage loading states and context seamlessly.

Ideal Generative AI vs. Reality

Foundational LLMs have read every byte of text they could find and their chatbot counterparts can be prompted to have intelligent conversations and be asked to perform specific tasks. Access to comprehensive information is democratized; No more figuring out the right keywords to search or picking sites to read from. However, LLMs are prone to rambling and generally respond with the statistically most probable response you’d want to hear (sycophancy) an inherent result of the transformer model. Extracting 100% accurate information out of an LLM’s knowledge base doesn’t always yield trustworthy results.

Chat LLMs are infamous for making up citations to scientific papers or court cases that don’t exist. Lawyers filing a suit against an airline included citations to court cases that never actually happened. A 2023 study reported, that when ChatGPT is prompted to include citations, it had only provided references that exist only 14% of the time. Falsifying sources, rambling, and delivering inaccuracies to appease the prompt are dubbed hallucination, a huge obstacle to overcome before AI is fully adopted and trusted by the masses.

Software developers use real-time data transmission to ensure the security of IoT applications. The choice of protocol is influenced by the complexity of the application and priorities. For instance, developers might prioritize speed over power saving if the IoT application requires real-time data transmission. On the other hand, if the application deals with sensitive data, a developer might prioritize security over speed. Understanding these trade-offs is critical to making the right protocol choice and putting in control of the IoT development journey.

As the Internet of Things (IoT) evolves, we witness the birth of the latest devices and use cases. This dynamic landscape gives rise to more specialized protocols and opens new possibilities and potential for innovation. Simultaneously, older, obsolete protocols are naturally phasing out, paving the way for more efficient and effective solutions. This is a time of immense potential and opportunity in the world of IoT.