What is the best way to manage git workflows for applications built on top of in progress libraries?

Question

I am doing something I haven't done prior, which is that I built a library (that is still in progress as it is somewhat ambitious), and now I am developing an application on top of the "alpha" version of my library.

As somewhat expected, as I am tinkering on the derivative application; I am encountering bugs/unexpected back end issues that I will have to adjust. Typically this hasn't been a real issue because it has only been one or two adjustments and then I go to my base library and push the new version and update the dependency in the derivative application I am making on top of it.

Right now though I have come across a spot where I want to introduce a whole new feature into my back end that I did not foresee needing initially. This is forcing me to jump back and forth between the back end, basically guessing what is the best way to fix it/implement it, then I have to push the change, update the dependency, and see if calling it in the front end of my derivative application works.

Obviously this isn't very good and it hasn't failed me entirely yet because I have been careful about what I am pushing; but it is embarrassingly inefficient and I don't know how best to handle this. Is there a more efficient way to develop applications on top of existing libraries in a situation like this? I have a sense that there are a couple things going wrong, and that I should mostly be relying on testing in order to ensure my functions work even before I push them; but I don't know how to write a test for something like this because in order to test the function I would need to mock a live data stream from a serial device and I don't even know where to start with that.

To give a specific illustration of what I am doing (*every time I need to make any change to the back end)

git stash whatever I am working on
git checkout main (the projects are in the same repo)
-make changes-
git add build.clj
git add ...back end files....
git commit ...
git tag...
git push...
-wait for it to build and publish-
git checkout -in progress branch of derivative application-
git pull origin main
git stash pop
-now I can see if my code worked, if not I do all of that again to try and fix my base library-

Seems pretty incorrect to me or at least something that can be improved upon but I don't know how. I have always felt a bit awkward using git and like it gets in my way more than helps me, and I know it is meant to make development easier but I get really stressed about it because lots of the things done through git are just permanent after they happen so I feel like I often do things that then make problems worse.

Mostly I haven't been able to find any help for this, but one person said "use worktrees", and while I realize it is on me to do research; I looked into it and I don't understand worktrees or how it would help me with this at a cursory glance because my front end still won't be able to use the back end code until I already have pushed and published the base code because even though they are in the same repo; they are completely different projects. The derivative work I am using is literally an example of how to use the base library and so I don't know how to best manage jumping back and forth between them.

• user1937198: If your frontend is pulling from the published code, why do you need to push the lib from the same checkout? Why can't you work on the lib in one checkout, publish that, and then pull from a second checkout?
• user1937198: Also, what build system are you using, and does it have the ability to repoint the lib depenency from the published version to a local build for development?
• Greg Burghardt: Can you delete your cross-post at StackOverflow? - stackoverflow.com/q/79708378/3092298

Answer 1

The way you are using git to manage your projects is indeed likely to get in your way. Especially for future projects you might want to reconsider your decision to have different projects on separate branches in the same repository.

But, given your current setup, here are some pointers to make your life easier:

1. It is no problem at all for git to have the same repository checked out multiple times on your computer, as long as they each live in a different folder. This means you can have the library checked out in one place (e.g. c:\git\library or ~/git/library) and at the same time the application in another place (e.g. c:\git\application or ~/git/application) and each of them referring to a different branch.

   This makes it possible to just have two editors open (one with the library and the other with the application) and work on them in parallel without the whole branch switching dance.

2. See if your dependency/build system has a mechanism to use a local build of a dependency instead of a published version. That should reduce your turnaround time when making a change to the library.

3. If using a local build is not possible, setup your build system such that a build of an untagged commit also gets published, as something like SNAPSHOT-<branch>. That also saves you from continuously updating the dependency in the application. The naming with SNAPSHOT is a common way to indicate it is an unstable build (either not properly tested or the contents may change without notice).