Design Question: Keep 1000 newest items; DB vs. application

Question

I have a Qt application, which sends and receives messages. Messages are stored in a local MongoDB.

The application has a message list window, which shows sent, received and all messages (depending on the view). For performance reasons the application has a QList, which holds QPairs of Mongo ID and the message itself, so more or less all messages are available in the application.

We want to limit the no. of shown messages, because the view gets slower for 1000 and more messages.

The simple approach was just to limit the no. of pairs stored in the QList.

However, I'm not quite sure, if this is true. Recently I've read here (EDIT: found it) a question which basically said "Your application code won't be smarter/faster/better for stuff a DB is designed for (e.g. sorting, getting a subset, etc.)". In addition keeping the QList in sync with the DB is causing us headaches and we have some corner cases like an very old message can be updated and then has to be displayed again.

So the question is, would a DB-only approach (e.g. querying the DB for 1000 messages whenever the view has to be updated) be faster then storing the acutally shown messages in the QList and trying to keep them in sync.

• rlperez: DB might be better but it depends on how far away it is from your machine and also if it makes sense to cache any of that data.
• Simon: DB is locally on the machine. We use the DB, because we have a secound application, which uses the messages also.
• Pieter B: Why does your view need 1000 messages? Does it show a long list of them all at once?
• Simon: Let's say 300 would probably be also sufficient. It's a list with timestamp, message type and first 30 chars of a text field. User can select a message and display it.

Answer 1

This seems more like a separation of concerns between the: QList, view and MongoDB.

Focus on the purpose of the QList and then make sure the synch capabilities get the job done. Do you need all the records for some purpose outside of the view?

We want to limit the no. of shown messages, because the view gets slower for 1000 and more messages.

If the QList for whatever purpose outside of displaying a view, needs to hold more than the 1000 view limit, you need something else to limit the records coming from the QList to the view (if that's how you need it to work.). Otherwise, you may need some other logic pulling the MongoDB data with a 1000 record limit with the sole purpose of feeding the view.

Is the limitation of the view (not sure exactly what that is) holding enough records in memory or the control you're using as the display for the user (Data Grid, Combo Box, repeater, etc.)?

• Simon: The view is basically a list with timestamp, type and 30 chars of a textfield of the message, which is provided with data by a QTableModel. The QTableModel holds the data in the list.

Answer 2

It's impossible to say whether loading from the database will be faster without running some tests or knowing how often the window gets refreshed. My guess is that the perceived performance loading or working in the messages window will become constant rather than getting worse as the number of messages increases.

A not-insignificant upside to it would be simplifying your code a bit - you'll no longer need any code to keep the list in sync with the database. You'll just grab the latest (perhaps configurable number of) messages when loading or refreshing the messages window, so you won't have to worry about stale data.

As you said in a comment:

Let's say 300 would probably be also sufficient.

Retrieving a very small amount of data from the database will very likely not be noticeably slower, and if 300 messages can cover 90% of use cases and you can provide a way for users to retrieve older messages when needed, I bet you'll come out ahead performance-wise.

• Simon: Our goal would be to simplify the code for maintainability and let the DB do most of the stuff. On the other hand the DB solution should be at least as fast as our application code, which is not too fast at the moment.