Architecture for low-latency RESTful server on Amazon EC2 with Node.js and a database

Question

This is a question for DBAs with a lot of practical experience with large scale deployments.

• We are trying to build an app with very low latency. (We really want to create the best user experience, and latency matters in this case.)

• The app is fundamentally a RESTful Node.js server that queries a database (OrientDB). Static content/files are hosted elsewhere. Objects are translate into UI largely on the client end.

• We are planning on using Amazon EC2 to host the web server and database.

• High availability is important as well.

• Cost is not a major deciding factor at this time.

What architecture would you recommend?

Here are 2 options, maybe there's more:

• Put the webserver (Node.js in this case) on the same EC2 instance as the database. We'd have multiple instances like this for availability and load balancing. In the past, I've used this to reduce latency in high volumes but those system didn't have to scale as dramatically. It was also before placement groups.

• Or, have databases on different instances than the REST web server.

Background

The database size is about 50 GB and we need a better solution. Projections look like traffic will be 10x that amount (a 500 GB DB) within 8 months.

It is about 20% writes and 80% reads. There's about 1000 queries per second right now and the rate is increasing.

• user2501: Why would you not ask this n Database Administrators that's where all the real DBAs hang out.
• SilentSteel: Whoops, I didn't know dba.stackexchange existed. I flagged this post to have it moved.

Answer 1

"Put the webserver (Node.js in this case) on the same EC2 instance as the database"

Never put your app server and DB server into one machine. Since you are using EC2 there are some white papers on Amazon named "Architecting on AWS" and there is a section called scalability. All your app servers should be state-less. It means it should be no data stored on any machine so you can easily bring a VM up (from your AMI) based on your auto-scaling group or get rid of it when the traffic goes down.

Database servers usually have their own way of scalabilities like Master-slave or replica-set. So they shouldn't be in the same group of your app servers (auto-scale or ELB).

Now the network latency, actually lately I was testing internal network speed check on AWS VPC and it wasn't that bad at all. Depending on your EC2 type the network has the low-moderate-and high type of network but when it comes to internal it's pretty much fast. It will be inside DNS resolving time, HTTP request time, HTTP respond time, etc. You should be more worried about your query time on EC2 machines by choosing SSD volumes (Provisioning your EBS volumes maybe) and maybe RAID your EBS volumes to speed up read/writes. The bottleneck usually is the I/O when it comes to the database in architecture, not the network latency (unless you are using all the bandwidth so you need more secondaries so your app servers can read from them directly).

Hit me if you have a question, Good luck

• SilentSteel: Thanks. You said stateless. I'd love stateless but wouldn't there be tons of overhead if we couldn't use sessions on the app servers? We have a huge install and I heard redis won't scale past a certain point. What are your thoughts on sticky sessions? I heard ELB supports that.
• Maziyar: @SilentSteel In Architecting on AWS they wouldn't recommend sticky sessions. For storing sessions there is a managed service by AWS named ElasticCache that has Redis so it will take care of the maintaining your Redis and it will be stable. I am using MongoDB to store sessions which is find for low traffic. Have you considered DynamoDB for storing your sessions?
• SilentSteel: Hmm.. those are good ideas- ElasticCache and Redis. I had a call with Amazon and they agreed with your advice. We could also retransmit from the client end. We will probably not use sticky sessions (or at least keep a copy of sessions in ElastiCache or something similar.)