in my company we are using different 3rd party systems, for example:
- a CRM, where we sync information about the customer; the goal is having a 1 to 1 syncronization.
- a transactional email system, with whom we can send email to the customer.
At the moment we are having several timeouts, because these systems have a fixed threshold of requests per second.
How do we handle these timeouts? For example, we stored the emails we can't send in our DB and a scheduled function retrieved them and tried to send them again.
We tried also to use a RateLimiter, but this will not work properly if you try to scale up multiple instance of the application.
But there is a proper way to handle these errors? It seems a really common problem, maybe there is a pattern to consider. These system doesn't support streaming or bulk API.
1 Answer 1
Consider implementing a Circuit breaker which will find when third party is available to retry request:
Assume that an application connects to a database 100 times per second and the database fails. The application designer does not want to have the same error reoccur constantly. They also want to handle the error quickly and gracefully without waiting for TCP connection timeout.
Generally Circuit Breaker can be used to check the availability of an external service. An external service can be a database server or a web service used by the application.
Circuit breaker detects failures and prevents the application from trying to perform the action that is doomed to fail (until it's safe to retry).
We tried also to use a RateLimiter, but this will not work properly if you try to scale up multiple instances of the application.And it never will because the bottleneck is on the 3rd party side, not yours. You have to dissipate your concurrency over the 3rd party service, scheduling the load. It also depends whether sync+email is a single transaction or both might happen at different times (so they can be treated as different transactions).