Can one database affect the performance of another in MongoDB?

Question

We have a fairly beefy server (15 GB RAM, 2 cores) running a reasonably large MongoDB instance. The Mongo instance hosts two separate databases that communicate via application code. The larger database (let's call it DB1) contains on the order of 100 million records with more being added all the time.

The second DB (let's call it DB2) contains about 10 million records and grows much more slowly (there are a lot of writes, but most are updates).

Recently, we noticed that reads on DB2 have become ridiculously slow, taking up to 2-3 minutes to simply loop through about a 1000 records. On profiling the query, it looks like the right indexes are set and are being used, but that the performance for some as-yet unknown reason has completely fallen off a cliff.

Considering that DB2 doesn't grow much, could this be because DB1 is affecting the performance (maybe by competing for RAM)? Is there a more direct mechanism for two databases to interfere with each other?

I'm trying to identify the bottleneck here, so any pointers in the right direction would be much appreciated.

I should also note that while DB1's performance has also degraded a bit, it hasn't completely fallen off a cliff like DB2.

• Stennie: What specific version of MongoDB server are you using, and if >3.0 what storage engine?
• Chinmay Kanchi: MongoDB 3.0.12 and we never changed the storage engine and we migrated (mongodump-restore) from 2.x. so I guess it's MMAPv1. But I'll have to check once I'm at work to give you a definitive answer.
• Stennie: What O/S and filesystem are you using and how big is your total data (in the dbPath)? Have you reviewed the mongod logs during periods of slowdown? There generally will be some hint in the server logs about slow operations; Asya's presentation on Diagnostics & Debugging is a great intro to tracking down performance issues. The MMAPv1 storage engine in MongoDB 3.0 has collection-level locking so there shouldn't be any obvious interaction between DB1 and DB2 aside from server/instance resource contention.
• Stennie: If DB1 is much more actively updated, perhaps your working set (in memory) consists of more pages from DB1, and as a result DB2 queries are slower because indexes or data need to be paged into memory.
• Chinmay Kanchi: Interestingly, just retiring a ton of data from DB1 improved performance on both DBs considerably. So I'm gonna say resource contention was the issue here.

Answer 1

I can add, that WiredTiger and mongo 3.2.11 will not help you. Theoretically it hasn't global locks, but in practice I have a bit more beefier server (12 core, 96GB RAM) and two databases. DB1 is in production (it is only 300GB in size, so mostly in RAM) and DB2 which I prepare for next release.

Surprisingly, found that mass updates and deletes on DB2 (also 300GB), even in one thread (one console or java application) make regular indexed queries on DB1 run for 5-10 SECONDS.

When I stop all activity on DB2 it returns to 3-400ms.

In other topics I've read that deleting by unindexed field might lead to such behaviour, but unfortunately, not in my case. I update/delete by "_id" and still performance is terrible.