How to do "initial sync" of data for new microservice that depends on data from another microservice?

Question

Hypothetical scenario:

Let's say I have a Customer service. It has 1 million customers in its database. I then create a new service that depends on a specific projection of the customer entity.

I want to avoid coupling and synchronous communication, so the new service should get customer updates through asynchronous messages.

But how do I sync the existing 1 million customers to the new service?

I've read suggestions that the Customer service should simply emit messages for all the customer entities, but there are a couple of problems with that solution:

1. The new service needs all the data from the start. Failure because a needed customer isn't synced yet it not acceptable.
2. Consuming 1 million messages is slow. I admit that I might have a suboptimal implementation here, but having Azure Functions trigger on messages and write updates to Azure Sql Database will quickly consume connections and hit the limit, unless properly throttled. Throttling slows the sync down and exacerbates point 1.

Is there a better way?

I'm thinking some kind of tool that writes directly from database A to database B, while mapping the model from A to B.

General suggestions and suggestions specific for the Azure context are appreciated!

Edit: what if the new service needs to sync data from two or more services?

• Thorbjørn Ravn Andersen: backup + restore?
• AleksG: Yeah I guess that could work! Then change to desired schema after the fact. Or maybe one could change schema before restore?
• John Wu: For this you should probably do a database migration, which is a broad topic unto itself. If you are .NET then I suggest Data Migration Assistant as a starting point.
• AleksG: @JohnWu Thanks for the hint, this looks promising.

Answer 1

If this is a one-off process, a backup/restore type process is probably the best way to go. Personally, I would suck it up and go with the RabbitMQ message route myself. Its less prone to mis-matched schema problems I would imagine