How to Stream Sensor Data to Apache Pinot for Real Time Analysis

This is part 2 of a multi-part sample application that I've been building using Kafka and Apache Pinot. In the first part, we built an NFC Badge Reader to record when badges are swiped. In this part, we will begin ingesting some environmental data from a sensor into the mix. I'm not going to spoil the fun by telling you where this is ultimately headed, but feel free to reach out to me on Twitter and see if you can guess!

As always, all of the code is available on GitHub.

Real-Time Analytics for IoT

We’ve been hearing that the Internet of Things (IoT) would transform the way we live and work by connecting everyday devices to the internet for a long time now. While much of the promise of the IoT always seems to be "coming soon," the proliferation of IoT devices has already created a massive amount of data that needs to be processed, stored, and analyzed, in real-time. I’ve said for years—actually over a decade now—that if your IoT data isn’t timely, accurate, and actionable, you’re mostly wasting your time in collecting it.

This is where the Apache Pinot® database comes in. Pinot is an open-source, distributed data store designed for real-time analytics. The high scalability, reliability, and low latency query response times of Pinot make it a great solution for processing massive amounts of IoT data. In this post, we will explore the benefits of using Pinot in IoT applications.

Getting Started with IoT and Camunda Platform 8

I've been writing about and doing, IoT for almost 20 years now and one of the things I am continually doing is seeing how I can use different technologies to further the state of the art in IoT. I've invested a lot of time and effort in seeing that IoT is successful, and finding innovative ways to look at IoT.

It turns out that close to 70% of IoT projects fail at the proof of concept phase. Why? because they are unable to deliver any lasting value to businesses and are thus abandoned. This is one reason why I believe that, even though IoT has been "about to explode" for over a decade, it has yet to deliver as well. 

A Camunda Greenhouse: Part 3

Review

If you've been following this (incredibly slow) project, then you've already been through part 1 and part 2 and have been wondering if part 3 would ever arrive. Well, here it is! This is the part where I pull together all the previous parts and actually deploy a Camunda BPMN-powered IoT Greenhouse!

Just to review a bit, I proposed building a greenhouse, monitored by some IoT sensors, that would then be entirely controlled by a Camunda BPMN process. The entire process would then control all aspects of the Greenhouse.

Automating IoT With Camunda Platform

Some Background

When I first started at Camunda back in October 2020 (what was 2020 anyway?) the very first thing I was asked to do was come up with something I could do for a special Halloween blog post. It being COVID-times, I of course built a Camunda and IoT integration to evaluate costumes and deliver candy.

I am sort of well-known for doing weird, pointless IoT projects like this one and this one, and this one. You get the idea.

Camunda IoT Proof of Concept: Part 2


This is Part II in the series covering a Proof of Concept (PoC) project I’m working on as part of my job as Principal Developer Advocate at Camunda. I’m not sure how many posts will be in the series, but, well, at least two? If you missed Part I, you might want to catch up on it before continuing.

This part of the project was the first hardware build of the project. I have now built and deployed a complete outdoor weather station to gather data about current weather conditions outside the greenhouse. This will allow me to compare conditions inside the greenhouse with conditions outside the greenhouse, and compensate accordingly.

Letters to Santa – Automating Joy to the World at Scale

It’s that time of year again. The time when the world’s largest order fulfillment operation experiences its heaviest load. No, not Amazon – we’re talking about Santa Claus, Inc. – the largest logistics company in the world, with a 24-hour global delivery window at peak load.

This year is different, however. Earlier this year, Saint Nick clicked on an ad on his Facebook feed, one promising a digital nomad lifestyle through automating his business. Sick of the Arctic weather and the stress of traveling, the thought of sitting on a beach in Thailand – while still bringing joy to children around the world – was enticing.

[CSF] Build Yourself a COVID-Safe Halloween Candy Dispenser Using Arduino and Camunda BPM

It's a very Camunda Halloween, and this project should keep you safe from the Trick-or-Treaters!

In this year of COVID-19, I needed to find a way to make Trick-or-Treating safe and fun for everyone. I knew it needed to be no-contact, but I also knew I needed a way to properly evaluate the costumes and Trick-or-Treaters to give out candy. What I came up with is a combination of IoT and Business Process Management (BPM) that allows you to safely evaluate Trick-or-Treaters from afar and dispense candy on demand! It's more than just a BPM process, of course. It's an IoT project that uses Camunda BPM to deliver the goods.

Sending IoT Data From Arduino to Questdb

IoT on QuestDB

As you can well imagine, I've been super busy in my hew job at QuestDB so this has taken me longer than I would have liked, but here we are. If you know me at all, you know that one of the first things I always do with new things is, well, connect things to them! So I immediately went to connect an IoT device to QuestDB to see how it went. Unsurprisingly, it went quite well. So here's how it went, from start to finish.

The Database Part

The first thing I had to do was to get QuestDB up and running. Luckily, this is very straightforward. I guess I could have gone the Docker route, but as you're probably aware, I'm not a huge fan of Docker (in no small part due to the fact that it will literally suck the life out of a macOS laptop). There's also (for you MacOS users) `brew install questdb` but since I work here, and I wanted to test out the latest and greatest web console, I decided to build from source:

Using Flux to Get IoT Sensor Metadata From MySQL

Learn how to use Flux.

If you've deployed an IoT solution, you've had to decide where, and how, to store all your data. At least from my perspective, the best and easiest place to store the sensor data is, of course, InfluxDB.

My saying that can't come as a surprise to you. What about the other data you need to store? The data about the sensors? Things like the sensor manufacturer, the date it was placed into service, the customer ID, what kind of platform it's running on. You know, all the sensor metadata.