How to design document exchange system

Question

I am trying to come up with a design of document exchange system. System should be for business-to-business communication. Description: There will be possibly 100+ clients(consumers) - they will have to develop clients independently. Clients should send and receive documents.

Message Data:

• Author client ID
• Recipient client ID
• Document data (title, body, etc)
• Attachments

client IDs known for everyone

Whole document/message might become 200MB+ or even GBs (depending on how many attachments are presented).

There should be other similar operations to sending and receiving documents, that should follow the same principle, so I'm not listing them for brevity.

What methods I discussed:

• REST/SOAP - but there is no way to push messages to clients, when someone sent to them (one way is that client could have scheduler to regularly check if they have some documents to receive, which I don't like because messaging won't be instant).
• AMQP - even though solves the first problem, it has some limitations, like when some client sends me a document I cannot respond to them. For example: validation errors and etc.
• REST/SOAP for only sending and AMQP for only receiving messages. This just complicates everything.

Looking for suggestions, that could make my life easier, spend hours and hours but nothing good comes up to my mind.

Note: Messages might be 200MB+ in size and possibly represented as XML or JSON

Example flow:

1. client->sendDocumentRequest->server->sendDocumentResponse->client

2. server->pushDocument->some other client

I hope I explained well, if needed any more information for clarification, please ask. Thank you.

• Laiv: I'm afraid of the question is fairly too broad since there are several ways to do this and probably none are REST, AMQP (you can bet), SOAP and the like. So far you have two main concerns to start working on 1 messaging 2 binary data transfer. both can be decoupled, there's no need to make everything in a single transaction/connection. They both also can happen at different times. Moreover, the server could have nothing to do with the uploading and the downloading of the files. Don't look at it as a single problem/single system.
• onlydmc123: Thank you for commenting! As I understand you are saying that I can decouple this to 2 other services: one for sending and receiving, and another one for uploading/downloading attachments (possibly ZIP archives, containing all of them, to avoid too many requests and maintain consistency of message easily when document for example has 50+ attachments). The latter service is easy to implement, but what about first one, should this be done using messaging (AMQP for instance)? or some other method?
• Laiv: 1st depends on the needs. it could be a mere Web API or something more sophisticated. It depends on the requirements and constraints of the system. 2nd could be a mere Sftp server. Or more sophisticated, you can wrap the sftp server with a service so that you can freely change the storage support anytime without compromising the clients. Let's say you start with an SFTP but then you move to a NoSQL data storage
• onlydmc123: 1) If the first one was Web API, clients should have scheduler to regularly check for incoming messages (which I'm trying to avoid). The needs are: instant messaging (well, except the data client has to download from second service) + when client sends a document it should be validated and responded to client. If messaging was a choice validation would be sent asynchronously and correlationId would match the sending document, but this complicates everything (specially for clients), as well as lacking the direct response to client when erroneous document is being sent. This is where I'm lost.
• Laiv: but this complicates everything (especially for clients) who said it was going to be easy? 2 concerns 2 solutions. Divide an conquer

Answer 1

I just wanted to grab onto the message size, having designed server software with similar requirements.

I don't know the exact technology stack you would use, but the message size excludes a lot of solutions out of the box. Basically you don't want that whole message to be kept in memory, because even 1 message might use it up completely, 100s of clients would be almost impossible without a massive installation. The solutions for avoiding using memory:

1. Non-blocking zero-copy solutions. These don't use up threads nor memory regardless of message size, therefore they are the best for these sorts of things. You can either take something off the shelf (complete file transfer solution), or implement on top of something like vert.x (for JVM languages).

2. Streaming. Things like Servlets or other containers usually support this. This blocks a thread, and copies data a couple of times, but can support any message size with little overhead. For 100s of clients (using up a couple dozen threads) it would probably work.

3. Messaging. This wouldn't solve any of your problems directly. You would still have to read messages and avoid the same problems as above.

• onlydmc123: Thank you for suggestions! I'm going to build it using Spring framework.
• Laiv: @RobertBrautigam Vert.x seems interesting. Have you implemented a web application with this framework? Is it based on the same principle as NodeJS? A single-thread on a event-loop?
• Robert Bräutigam: @onlydmc123 Spring falls into category 2. Still, that would probably be sufficient.
• Robert Bräutigam: @Laiv Yes, I used it for a web-application (for non-human clients), using Vert.x Web. It is using Netty under the hood, which is event based, but usually not single threaded, although it can be configured that way.
• Laiv: Sounds interesting! Even if it's not single-threat. I don't know Netty, but I will check it out. I would like to have alternatives for Spring Boot when it comes to implementing web solutions or mere client-server. Something lighter and if it supports event-driven development much better.