Edit - Software Engineering Stack Exchange

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

How should we design an IoT platform that handles dynamic device schemas and time-series ingestion at scale (100K writes/min)? [closed]

We’re a small dev team (3 full-stack web devs + 1 mobile dev) working on a B2B IoT monitoring platform for an industrial energy component manufacturer. Think: batteries, inverters, chargers. We have 3 device types now, with plans to support 6–7 soon.

We're building:

A minimalist mobile app for clients (React Native)
A web dashboard for internal teams (Next.js)
An admin panel for system control

Load Characteristics

~100,000 devices sending data every minute
Message size: 100–500 bytes
Time-series data that needs long-term storage
Real-time updates needed for dashboards
Multi-tenancy — clients can only view their own devices
We prefer self-hosted infrastructure for cost control

Current Stack Consideration

Backend: Node.js + TypeScript + Express
Frontend: Next.js + TypeScript
Mobile: React Native
Queue: Redis + Bull or RabbitMQ
Database: MongoDB (self-hosted) vs TimescaleDB + PostgreSQL
Hosting: Self-hosted VPS vs Dedicated Server
Tools: PM2, nginx, Cloudflare, Coolify (deployments), Kubernetes (maybe, later)

The Two Major Questions We're Facing:

1. MongoDB vs TimescaleDB for dynamic IoT schemas and time-series ingestion? We need to store incoming data with flexible schemas (new product types have different fields), but also support efficient time-series querying (e.g., trends, performance over time).

MongoDB seems flexible schema-wise, but might fall short on time-series performance.
TimescaleDB has strong time-series support but feels more rigid schema-wise.
Is there a proven pattern or hybrid approach that allows schema flexibility and good time-series performance?

2. How to structure ingestion for 100K writes/min while supporting schema evolution? We’re worried about bottlenecks and future pain if we handle ingestion, schema evolution, and querying in one system.

Should we decouple ingestion (e.g., raw JSON into a write-optimized store), then transform/normalize later?
How do we avoid breaking the system every time a new product with a new schema is introduced?
We’ve also considered storing a "data blob" per device and extracting fields on-demand — not sure if that scales.

Additional Sub-Questions: (Feel free to address any of these if they fall into your expertise area)

RabbitMQ vs Kafka — Is Kafka worth adopting now or premature for our stage?
Real-time updates — Any architectural patterns that work well at this scale? (Polling, WebSockets, SSE?)
Multi-tenancy — Best-practice for securely scoping data per client in both DB and APIs?
Queue consumers — Should we custom-load-balance our job consumers or rely on built-in scaling?
VPS sizing — Any heuristics for choosing VPS sizes for this workload? When to go dedicated?
DevOps automation — We’re small. What lightweight CI/CD or IaC tools would you suggest? (Currently using Coolify)
Any known bottlenecks, security traps, or reliability pitfalls from similar projects?

We're still early in the build phase and want to make smart decisions upfront. If any of you have dealt with similar problems in IoT, real-time dashboards, or large-scale data ingestion — your advice would mean a lot.

Thanks!

Answer*

> We need to store incoming data with flexible schemas (new product
> types have different fields), but also support efficient time-series
> querying

That's the neat thing, you can't!

The essential problem is that schemas enable fast and coherent queries. Just because you can store a json blob with no schema, doesn't mean your reports will run fast or continue to work when the data structure changes.

Also, though 100k writes per second is fairly low in the scheme of things, you can easily imagine problems as you scale.

You have already mentioned the general solutions to these issues

1. Split ingestion into steps, Ingest whatever is sent and then have a second post processing layer which adapts the data into a reportable schema.

 This gives you a protection/anti corruption/Extract Transform Load layer allowing you to deal with schema changes and having multiple schemas live at any given time,

2. Shard the database. 

 You will want to allow the ingestion to branch out, try to collect as locally as possible and package up the data to keep the bandwidth down. But also allow for sharding by product and tenant to protect your system from scaling issues

3. Reporting/Querying

 100k data points are more than you can display on a normal graph even if you can ingest them. Add another transform layer which rolls up metrics into averages and totals for fast reporting.

If you do these things it can work well, but you have to abandon to some extent the idea that you can throw any form of data at the system and do a quick change to the report. Which is the feature things like splunk or elastic promise.

You have to plan and understand changes to the schemas, and add transforms and indexes to support your queries.

Sub questions:

These are a bit too detailed to go into, but I think you should check out some of the off the shelf ETL systems before you pick a language for your backend. This wheel has definitely been invented and the product you choose will determine what language you write in to some extent.

Draft saved

Draft discarded

Edit Summary*

Cancel

Add a comment |

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. design-patterns), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »