Why data export is crucial

There’s a common thread running through almost every Connhex functionality: a feature that seems obvious has some traits that make you go “actually, I never thought of that”.

We released Connhex Exporter, an improved way of exporting data from Connhex: this post shares some highlights on the hows and whys.

Why exporting is important

Being able to export data is an essential requirement for any IoT platform. Two obvious reasons behind this:

• data transfer: would you ever use a software service knowing you would be forced to keep using it just to browse through your data? Me neither.
• further processing: as much as we like to design feature-rich UIs, every once in a while all you need is a CSV dump that can be imported into Excel.

These are important factors, but that’s not all. The real case for data export being essential is another one: it is a foundational feature - more on this below.

Why all the fuss? Isn’t it just an export?

The problem here is performance: remember, we are usually dealing with hundreds of millions of rows in a single database cluster.

Not every Connhex instance will have databases with read/write splitting, so we need to make sure the ingestion pipeline’s performances are not hampered by exports. Also, suppose each user can create many exports at a time¹ : how do you define a balanced policy between a FIFO logic and a minimisation of the median wait time for each user?

It’s nuances like these all the way down.

How Connhex Exporter works

Connhex Exporter is a service designed to export data stored in Connhex Cloud. By data we only intend information coming from devices: additional details (e.g. user profile, personal information, …) are managed separately, mainly due to completely different technical requirements.

We decided from the get-go it would need to be:

• data-agnostic
• asynchronous
• scalable

By operating asynchronously, Connhex Exporter allows for the execution of resource-intensive operations (i.e. querying lots of data). Here we cycle back to the foundational feature discussion of before: we have already built several functionalities on top of the exporter. A few examples:

• bulk exports
• reports
• recurring exports (“export these 5 metrics every 15th of the month”)

Connhex Exporter seamlessly integrates with Connhex Notifications: just select the interest period and metrics - the export result will be delivered through your notification method of choice.

Learn more

You can find detailed docs here: you can also subscribe to our newsletter to get every post delivered straight into your inbox.