Understanding IoT Data

Blog - Understanding IoT Data

puzzle2_bwWhat if I wrote this entire message in some esoteric language that you didn’t understand?  You would be able to see the words and understand that it came as an email or was in a blog post. You would be able to forward the message to somebody else. You could even store it somewhere and retrieve it later if somebody asked you. But would you intrinsically know how to respond to the contents? Would you know how to query what’s in it? Would you have any idea why you received the message?

At MachineShop we’ve invested a lot of time and energy into our message ingestion engine. We’ve used the Samza framework born out of LinkedIn and centered on Kafka to give us the persistence, fault-tolerance and adequately-decoupled broker-based stream processing necessary to drink from the IoT firehose. We can consume TONS of data carried on lots and lots of transport protocols like HTTP, MQTT, TCP, UDP, COAP…you name it…even email. Inbound messages are deposited into Kafka at the soonest possible moment to make sure we don’t ever drop anything on the floor. Then the magic happens. That’s when the translators take over, peeling messages off of Kafka in real-time to translate from proprietary protocols into good ol’ standard JSON. We know that as important as it is to be able to receive messages on all of those protocols at mind-boggling volume, it is way more important to understand the contents of the messages received - context is everything. After a message is translated, we have a standard way to start inspecting messages to run against rules that trigger alerts. You can start to develop apps across a myriad of platforms against a standard set of RESTful APIs. Everybody speaks the same language. No agents installed on your stuff. All in the cloud or in your intelligent edge appliance.

So, back to our unrecognizable message. Wouldn’t you much rather have your messages arrive in a language you understand? Doesn’t that seem more valuable? More actionable? And that is the idea behind our translation services. We unify data in real-time regardless of the capabilities of the data source so that you only have to speak one very universal language. Put a REST-ful interface on that translated data and all of a sudden your “things” look less like “things” and more like API endpoints.