How to manage common library dependencies to prevent cross-application breakage?

Question

Note: The question was heavily rewritten after the comments it received.

I work on a team that develops embedded software for various in-house devices. Each device hosts multiple "bare metal" applications—all compiled into self-contained binaries—and we also maintain a set of common static libraries (e.g., for networking, cryptography, etc.) used by the applications. All the code lives inside a single per-device repository where every commit goes into a main branch.

This setup has led to a major problem:

• Team members that are working on one application are breaking others application and work by committing changes to the common libraries.
• Developer of "App B that uses Lib A" has no way to choose a fixed version of the "Lib A" for his release.
• If developer of "App A that uses Lib A" needs to make a required breaking change it has to wait for the other developers to be ready before doing it's commit.

The codebase currently does not support automated testing. Making the code testable requires heavy refactoring that in the current situation we can't easily do because of the above It's something we are aiming to do after resolving or making this issue less problematic.

What strategy can we adopt to resolve the above problem?

• Arseni Mourzenko: There are quite a lot of issues with your question. The issues include, but are not limited to, the fact that (1) it is too large, (2) it asks for tools recommendations, which is off-topic, (3) it invites for a discussion, like it a forum, rather than answers, and (4) it includes irrelevant subjects, such as multiple vs. single git repository. In order to have a better grasp at how StackExchange sites work, make sure you skim through a few other questions, to see what's welcome and what's not, what gets upvoted, and what receives downvotes.
• Greg Burghardt: We have a limit of one question per post. Questions do best when you identify a tightly-focused problem to solve. There is nothing wrong with detail — we like detail — we just need the post to be more focused.
• titanicsnake: Thank you for the comments. I will try to heavily change the question so that it's focused on the main problem. @DocBrown The proposed solution was not there to make any advertisement or polititics but to share an IDEA of solution I thought about.
• Hans-Martin Mosner: VTR - I think the question as rewritten is focused enough and answerable now.
• Basilevs: The original was fine too!

Answer 1

Unfortunately, if you make a number of essentially independent codebases dependent on the always-latest version of a common library, then you get two problems.

One, is that any change to the common library requires far more extensive analysis to determine whether the change is breaking or not towards the dependencies.

Two, the scale and complexity of the above analysis, and the rework of many dependencies that may be entailed all at once, means a relatively conservative approach must be adopted when modifying common libraries.

A traditional justification for common libraries is that improvements to the common code are automatically incorporated into all dependencies, but this is a fallacy.

Many (not all) improvements require a global/whole-system readjustment of programming, rather than the changes being local to the common library and the benefits propagating automatically. Determining whether and where such sympathetic changes are required across the system, is often the majority of the effort - the changes themselves may be relatively trivial to apply.

And regressions, or faults due to missing-but-necessary readjustments, propagate just as automatically as certain improvements might.

Sometimes copy-and-paste incorporation of common code, actually contributes to better management overall, precisely because divergence is the default and it avoids the entire system of dependencies having to be analysed at once for every change.

However it is often not clear in advance which approach will be the better one, and neither approach will suit all scenarios.

What is important is to acknowledge that both approaches may be required at certain times, and ensure that it is possible to take either approach as the circumstances require.

So if you link to a common library, you need on occasion be able to switch to linking to a fixed version or a specific branch, rather than always to the very-latest version. Or if you copy-and-paste (or start linking to fixed separate branches), you need a system for tracking each place where a copy is used, and control any unintended or excessive divergence between each place.

• titanicsnake: Thank you for your response. Its surely me but I'm having some difficulty finding the actual answer in it to the problem I proposed. To me it looks like a more detailed explanation of the problem that I have (details that they asked me to remove from my query because the post was too detailed..) Maybe I'm missing the point!
• Steve: @titanicsnake, it's in the last paragraph. If you link to common code, you need a facility to link to a specific branch or version. What that looks like depends on your build toolchain. Or incorporate a separate copy-and-paste version of the code - that solves your immediate problem outright, and it may or may not create a less immediate maintenance issue.

Answer 2

There isn't a purely technical fix to this. The projects will need to communicate and coordinate changes to the shared libs.

It's something we are aiming to do after resolving or making this issue less problematic.

That's backwards thinking. If you had test suites for all your applications, then each shared library can gate changes behind the tests being run. That in itself will make this issue less problematic, at least to the degree that Dev A will become aware they are breaking App B.

All the code lives inside a single per-device repository where every commit goes into a main branch.

Do you mean all the code, or only the per-device code? Dev A can't break App B if they don't commit to a repository used by App B. Does each App have to pull the HEAD of shared code? You can instead pin it to a known version, and periodically advance a given app to the current HEAD (or a known stable point) at a time convenient to that project.

• titanicsnake: I'm aware that tests would make changes less problematic and that's why I've written that sentence. Problem is that the current code is impossibile to test since it depends heavily on running on real hardware. Eliminating those dependencies would require a major refactor that is impossible because of the problem described in the post.
• titanicsnake: ALL the code for ALL applications of a device reside in a single repository. Each device has a single repository. There is a lot of code that is common between those devices and that code is an exact copy in each repository. Since for a device all apps and shared code lives inside a single repo currently we have no way of "pinning" an app to a specific version of the shared code
• Caleth: "impossibile to test since it depends heavily on running on real hardware" isn't true. You can still gate changes to the repo behind running tests on real hardware, it's just more inconvenient than if you didn't need real hardware.

Answer 3

Libraries seem to need independent milestone/feature versions

Minor changes like localized bug fixes are trivial. Breaking changes should deliver a new product, with test code, an explanation on the changes and what to do to migrate.

I might imagine, that here the organization is too isolated seeing the nature of embedded software development. Causing the changes not to be packed in large version steps, but with hacking small code changes to a library till one application works.

The solution is tagging larger breaking steps, probably making it a new library major version. With documentation, how-tos, test code. And especially with a list of using projects of this version.

Thus one can plan the migration of the version. How many versions one is back for this application, what are the benefits / improvements, how invasive are the changes, testing effort needed.

So instead of a amorphous stream of library versions, real mile stones. Organisatorical with as most budget feasible, documentation, peer-review, presentation (slides).

An example: there is a requirement to read a text. In a change request one wants to first try reading the text in full Unicode, with encoding UTF-8. When failing using the local platform encoding. Developing however it seems best to change reading an URL into buffering the text first as bytes. At this moment a regular minor version update becomes a major milestone update requiring more effort for documentation, unit tests and such. Also communication.

• Basilevs: Yep, this problem has been solved with versioning multiple times. Linux package managers, NPM, Maven, SemVer people have been there and had success. It is not ideal, but works well enough.

Answer 4

The codebase currently does not support automated testing. Making the code testable requires heavy refactoring that in the current situation we can't easily do because of the above It's something we are aiming to do after resolving or making this issue less problematic.

What strategy can we adopt to resolve the above problem?

By refactoring the codebase to facilitate automated testing.

You won't make this issue less problematic until you do. I'm sorry, but I need to toss out a frame challenge on this one. You are worried about productivity or stopping another team's work in order to do this refactoring, but the reality is you already experience regular stoppages of work. Where is the line?

You don't need to stop all work on all apps on all devices. Focus on refactoring the shared libraries first. Make the shared libraries testable in small pieces. I can't recommend concrete techniques because I would need access to your codebase — something impossible for you to do. Instead, this needs to be a group effort with a shared understanding of the challenges and solutions.

Some additional ideas:

• Identify parts of the library that see the most churn and prioritize those first.
• Use feature toggles or compiler flags to turn code on or off. This can quickly become a big ball of mud unless you plan carefully.
• Changes to these shared libraries need to be coordinated better. Each device might need its own mini change control board for changes to shared libs.
• Can you write emulators for the hardware? This would at least allow you to write a test suite in software, but the complexity of setting this up would need to be balanced with the cost of setting up an automated test suite with real hardware.

I just don't think you can make this situation tenable until teams communicate changes better, and these libraries are changed to make them isolatable for testing purposes. It will cause things to slow down, but introducing automated testing for these shared libraries is the only answer long term, and will pay dividends down the line. You should see breaking changes causing work stoppages less frequently, which will free up more time to continue evolving the codebase and team process to finish alleviating these problems.

---

Another thought just came to me, but this requires using Git for version control. If you don't use Git, this might be just the incentive to make the switch.

Git Submodules.

A Git submodule is a git repository inside a git repository. It's a folder inside a git checkout that contains another git checkout from another repository. Make the shared libs a separate Git repository from the device repository. This allows developers to create branches and commit to these shared libraries. You will need to integrate changes and coordinate with other app teams, but this at least gives you the isolation necessary to apply changes in a controlled manner.

Failing this, you're left with my frame challenge above, and Git submodules might give you just enough flexibility to do the frame challenge in the first place.

Answer 5

1. Once you get to a certain level of maturity, stop making breaking changes without talking to the other team members. Set up some kind of "approval board", even if that's only the application team leads
2. Develop on the trunk but make tags (or whatever they are called in your version control system) once you are at release candidate status.

Answer 6

Move the responsibility. If developer X needs changes to the library that he needs for his app A and these changes break B and C, then make it that developers job to fix B and C.

Why? Because otherwise this guy is seen as the hero who creates new libraries at speed while the others are seen as lame developers who spend their time fixing broken code instead of doing anything useful. So guess who gets a raise. He has no reason to work more careful until he pays the cost of his carelessness.

Meanwhile do code reviews. You would check if your app works with the new code, otherwise the change is rejected until the library works with B and C just fine. Alternatively, remove the changes to the library if that is needed to make B and C work. Again, the work is moved to the person who checked in broken code.

You may also consider changing your build systems so the apps use a specific version of the library instead of the latest.