Edit - Software Engineering Stack Exchange

You are not logged in. Your edit will be placed in a queue until it is peer reviewed.

We welcome edits that make the post easier to understand and more valuable for readers. Because community members review edits, please try to make the post substantially better than how you found it, for example, by fixing grammar or adding additional resources and hyperlinks.

Required fields*

Rev

Required fields*

How should we design an IoT platform that handles dynamic device schemas and time-series ingestion at scale (100K writes/min)? [closed]

We’re a small dev team (3 full-stack web devs + 1 mobile dev) working on a B2B IoT monitoring platform for an industrial energy component manufacturer. Think: batteries, inverters, chargers. We have 3 device types now, with plans to support 6–7 soon.

We're building:

A minimalist mobile app for clients (React Native)
A web dashboard for internal teams (Next.js)
An admin panel for system control

Load Characteristics

~100,000 devices sending data every minute
Message size: 100–500 bytes
Time-series data that needs long-term storage
Real-time updates needed for dashboards
Multi-tenancy — clients can only view their own devices
We prefer self-hosted infrastructure for cost control

Current Stack Consideration

Backend: Node.js + TypeScript + Express
Frontend: Next.js + TypeScript
Mobile: React Native
Queue: Redis + Bull or RabbitMQ
Database: MongoDB (self-hosted) vs TimescaleDB + PostgreSQL
Hosting: Self-hosted VPS vs Dedicated Server
Tools: PM2, nginx, Cloudflare, Coolify (deployments), Kubernetes (maybe, later)

The Two Major Questions We're Facing:

1. MongoDB vs TimescaleDB for dynamic IoT schemas and time-series ingestion? We need to store incoming data with flexible schemas (new product types have different fields), but also support efficient time-series querying (e.g., trends, performance over time).

MongoDB seems flexible schema-wise, but might fall short on time-series performance.
TimescaleDB has strong time-series support but feels more rigid schema-wise.
Is there a proven pattern or hybrid approach that allows schema flexibility and good time-series performance?

2. How to structure ingestion for 100K writes/min while supporting schema evolution? We’re worried about bottlenecks and future pain if we handle ingestion, schema evolution, and querying in one system.

Should we decouple ingestion (e.g., raw JSON into a write-optimized store), then transform/normalize later?
How do we avoid breaking the system every time a new product with a new schema is introduced?
We’ve also considered storing a "data blob" per device and extracting fields on-demand — not sure if that scales.

Additional Sub-Questions: (Feel free to address any of these if they fall into your expertise area)

RabbitMQ vs Kafka — Is Kafka worth adopting now or premature for our stage?
Real-time updates — Any architectural patterns that work well at this scale? (Polling, WebSockets, SSE?)
Multi-tenancy — Best-practice for securely scoping data per client in both DB and APIs?
Queue consumers — Should we custom-load-balance our job consumers or rely on built-in scaling?
VPS sizing — Any heuristics for choosing VPS sizes for this workload? When to go dedicated?
DevOps automation — We’re small. What lightweight CI/CD or IaC tools would you suggest? (Currently using Coolify)
Any known bottlenecks, security traps, or reliability pitfalls from similar projects?

We're still early in the build phase and want to make smart decisions upfront. If any of you have dealt with similar problems in IoT, real-time dashboards, or large-scale data ingestion — your advice would mean a lot.

Thanks!

Answer*

> MongoDB vs TimescaleDB for dynamic IoT schemas and time-series ingestion?

Both are very capable to handle this sort of load. Note that you have one aspect really handy, I quote: “multi-tenancy — clients can only view their own devices.” This means, essentially, that **in this particular case, you can scale horizontally with ease**. Say you have a farm of five servers handling 1,000 users. After a big marketing event, you know that the number of users will double. No problem: you'll now need five extra servers.

> How to structure ingestion for 100K writes/min while supporting schema evolution? We’re worried about bottlenecks and future pain if we handle ingestion, schema evolution, and querying in one system.

I don't see an actual problem here. Don't worry *now* about the bottlenecks you *might have at some future moment*. That's **premature optimization**. **Measure what you have now, identify the *current* bottlenecks, if any; solve them.**

Since you can scale horizontally, the likely issue you'll encounter is at the network level, if you are unlucky to have an on-premise solution. Right now, it's not a problem, as you only receive about 500 KB/s—that's *nothing* by today's standards. But if it starts growing (for instance if the messages you receive start containing more and more information), then it could become a problem, forcing you to move to cloud (or a more capable data center). But, again, that's not what you should be focusing on *right now*.

> How do we avoid breaking the system every time a new product with a new schema is introduced?

By allowing the schema to be flexible. That's what you *already* did by choosing a database such as MongoDB.

> want to make smart decisions upfront

This is not how software development works. **You should postpone the decisions until the very last moment, not the other way around. This also means you should make it both *possible* and *easy* to switch from one thing to another.**

Practical example: I start writing a piece of software that would need to store some data, but I'm unsure how the data will be structured, and whether it would be better to store it in PostgreSQL or MongoDB.

Instead of spending my time *speculating* about which one is better, I would keep it easy: start to draft the piece of software, and store everything in a JSON file on disk, while *knowing* that I *will* have to switch to another storage thing later on.

The fact of knowing that would mean that I'll put in place the abstractions which will help me do the switch.

At the same time, I would release the MVP earlier, as I didn't spent time provisioning any database—storing everything in a plain JSON file *is* easy.

Now, I have an MVP that provides me with data about the actual usage patterns and the different storage issues I should address. With this data, I can make a concrete decision about the database I would switch to. No more speculation here. Or if I have doubts about the two databases, I can always draft two implementations of the database layer, and test them.

If, later on, it appears that I made a mistake, or circumstances changed, I can always swap the database layer for another one. No big deal.

> If any of you have dealt with similar problems in IoT, real-time dashboards, or large-scale data ingestion — your advice would mean a lot.

Not really.

You see, I do have real-time dashboards and do handle IoT data. I can write a post about how I do it, and explain why I'm using C++, or plain text files. This would be completely irrelevant for you, as my project is addressing the quirks that exist in *my* context (such as hosting the dashboards on Raspberry Pi 3), and not the ones *you* will encounter.

Similarly, you could find how large companies are solving this problem. But here again, different contexts require different solutions. Maybe some project developed by Boeing to gather the metrics from all the robotics involved in assembling an helicopter *look* very close to what you do. But the fact that they have chosen Fortran and Oracle may have to do with their corporate policies that you do not have.

Draft saved

Draft discarded

Edit Summary*

Cancel

1

"You should postpone the decisions until the very last moment, not the other way around." Well said. Flexibility is the smart decision, and that means waiting if you can.

Neil
– Neil

08/06/2025 08:45:32
Commented Aug 6 at 8:45

Add a comment |

How to Edit

Correct minor typos or mistakes
Clarify meaning without changing it
Add related resources or links
Always respect the author’s intent
Don’t use edits to reply to the author

How to Format

create code fences with backticks ` or tildes ~
```
like so
```
add language identifier to highlight code
```python
def function(foo):
print(foo)
```
put returns between paragraphs
for linebreak add 2 spaces at end
_italic_ or **bold**
indent code by 4 spaces
backtick escapes `like _so_`
quote by placing > at start of line
to make links (use https whenever possible)

<https://example.com>

[example](https://example.com)

<a href="https://example.com">example</a>

formatting help »
answering help »

How to Tag

A tag is a keyword or label that categorizes your question with other, similar questions. Choose one or more (up to 5) tags that will help answerers to find and interpret your question.

complete the sentence: my question is about...
use tags that describe things or concepts that are essential, not incidental to your question
favor using existing popular tags
read the descriptions that appear below the tag

If your question is primarily about a topic for which you can't find a tag:

combine multiple words into single-words with hyphens (e.g. design-patterns), up to a maximum of 35 characters
creating new tags is a privilege; if you can't yet create a tag you need, then post this question without it, then ask the community to create it for you

popular tags »