Monolithic Database to Microservices, but with Radical Data Model Changes

Question

We are finishing up a long (over 1 year) project to retire our monolith. We have been moving pieces of functionality into microservices guided by domain driven design.

So far we have been leaving the data model unchanged (we have been moving the tables owned by microservices into their own schemas). My question revolves around redesigning the data model, i.e the table and columns. Many of the core tables were designed around 15 years ago and the business even did a pivot in that time period. As you can imagine some of these tables are very far away from an ideal design, everyone is in agreement that radical changes to the data model are needed. Where there is not agreement, is in how to implement the changes.

There are 3 main approaches being considered:

1. Now that the tables are in their own schemas, evolve the tables incrementally until the "to be" design is realized.
2. Start with a green field design - new tables using current best-practices and current business requirements. Update the microservices so that they can work with either data-model. New creations would only be in the new model, but updates would go into either model (consider that the syntax of the primary key could tell you which model the data lives in). The data in these tables has a lifespan, so it would take a few years, but eventually the old data model could be retired and code supporting it removed.
3. Start with a green field design, but instead of modifying the codes to support both models, migrate the data to the new model before it goes live.

The main supporting arguments for each approach:

1. Lowest risk approach
2. Fastest path to "to be" design without the issues of data migration (business is 24/7 and some of the involved legacy tables have hundreds of millions of rows).
3. Fastest path to "to be" design without the complexity of split persistence layer in services.

Is there a clear winner among these choices? Can you point to any references related to a decision like this?

• Luc Franken: You have put them into separate schema's, are they really non-connected anymore or are services still picking data from other schema's? If a microservice has its own data schema and is locked off you can start migrating per-microservice (which is kind of the advantage of microservices). If not, if you choose to go all in for microservices, that would be the most logical main starting point. To create clear API's so the microservices can be upgraded independently.
• JimmyJames: I'm not sure I agree that #1 is the lowest risk approach. If I understand correctly, that requires many small data migrations. That's a lot of chances for data corruption which is, in my experience, the worst possible outcome.
• sceaj: @LucFranken - We're getting the DBAs "used to the idea" of not having referential integrity in the database, so the constraints are still there. As we move forward, we'll be migrating the microservice schemas to separate database instances. We are at the point where the APIs have been broken out but as I indicated, the persistence layer is far from a clean design for the domains they serve. We can do the migrate for each schema independently, but the main question is what strategy to use to keep the plane flying while we totally redesign it.
• sceaj: @JimmyJames - I'm sure that there would be mistakes along the way with #1, but if you consider risk of having a production outage, the incremental approach is safest because if something goes wrong, it is much simpler and faster to rollback than the other approaches which by design require more of a big-bang release to production. #3 has the added challenge of migrating while production is up and running.
• JimmyJames: @sceaj Yeah, when I first read your options, my initial reaction for #2 was revulsion. It's a lot of work. Once I saw the 24/7 requirement, I think it's your best option. I've been through an experience where database refactoring resulted in data corruption that wasn't noticed for a while. I don't wish that on anyone. The advantage of #2 is that you can 'try before you buy'. If something goes wrong, you have the old model and can fix it. Once you are certain things are good, you can commit to the new model.

Answer 1

You've already analyzed the possible approaches, including their justifications. All that's left to do is to decide which decision makes the most sense in your business context.

Internet strangers cannot decide this for you.

• sceaj: I understand that in the end we must decide. But I consider this a high-impact decision and cost of making a bad choice could be very high. I'll interpret your answer as at least validation that you agree were looking at reasonable options and correctly assessed the rationale for each. I haven't found much material on doing radical data model changes in live production systems, so I'm hoping someone can point me to a reference I have missed, or give some real-world experience.
• sceaj: Also, aren't Neal Ford, Robert Martin, Mark Richards, Martin Fowler, etc, "just guys on the internet"? Granted, they aren't strangers (any longer). :-)
• Flater: @sceaj Neal Ford, Robert Martin, Mark Richards and Martin Fowler are not telling your company what risks to assume and how to run your company. The issue here is that you're giving the decision to someone whose experience you've hardly vetted if at all, does not have the necessary context and does not suffer the repercussions of a bad decision. In other words, someone who has no skin in the game whatsoever. Letting such a person call the shots would be an act of egregious mismanagement.