What is the current definition of a Mono-Repo

Question

Facebook famously made a post about using a "monorepo" and for a while it seemed like this approach would take over from standard, separate out the code into multiple source controlled "repositories". but there has never really been much information made public about the actual approach used.

Previously we have had this question : Why does Meta (Facebook) use mono-repo in their source control?

To which I gave the answer that; to the best of my knowledge and previous investigations "monorepos" were really just attempts to solve problems with Perforce style source control where branching and merging have issues you don't see (as much) in git, using quite complicated custom build tools and conditional compilation of parts of the solution. (0 votes)

Where as, at least at the time, I found people using the term to mean, "just put all the code in one git repo and build it". An approach I think Ben Cottrel, sums up in his comment:

Separate repos are only a good idea if it results in true isolation. I have worked on too many code bases where a simple behavioural change to a requirement has involved changing 10+ different git repositories, each with their own versioning, tests, interfaces and pipelines, therefore requiring separate PRs, all to happen in the correct order; essentially turning a 30-minute software change into something which takes all day. Mono repos are preferable to this

This is fine, and you can see there is a balance to be made about when and what to split in any solution. But if mono-repo just means "don't prematurely split a repo" it's not a revolutionary thing to blog about.

Has the idea of monorepos taken off? what does the term mean these days, has it solidified into a single approach or competing ones and is it compatible with git?

Please answer with an explanation of the idea, the problem it solves, how it is implemented and what situations it is applicable to?

• Basilevs: Is not it just a way to deal with dependency hell? If any change in a component is accompanied by relevant changes in dependant components, there is no need to manage dependemcies.
• Basilevs: To generalize, mono-repo is a way to change the whole world atomically and independently of other changes. This reduces the need for coordination, which helps extremely large teams.
• Greg Burghardt: Wikipedia actually has a decent write-up. I don't think it fully answers you question though.
• Doc Brown: I upvoted this question (mostly to counter the usual unsolicited, unexplained downvotes which gnat and others always cast to almost any interesting question). However, after reading it again, it looks like a request for a list of case studies (especially the last sentence). The literal answer to the question's title is IMHO "the definition of a mono-reop is the same it was 10 years ago, the one you can find in Wikipedia". This leaves me a bit puzzled if this is really a good question for this site.
• Ewan: Its hard to phrase a question to be both open to unexpected answers and also tight enough to avoid the impression that I want a list, or discussion.

Answer 1

I have worked with mono-repo's across several different companies and SCM tools (Git, Perforce, Subversion).

Git

Assuming that all teams use a common branch name typically "main" it was easier to find code in "obscure" projects that I hadn't worked on before. I could just checkout "main" then either browse around the directory tree looking for likely suspects or do a recursive grep.

Teams also created "potential" anti-patterns such as creating branches like inventory-dev, inventory-feature1 and tags such as inventory-1.2.3 it makes it easy for them to find/merge their changes, because they only see commits to "their stuff", but if you need to build multiple products from different teams you need to check out different branches, build then switch to a different branch/tag. Additionally a graphical view of the history looks like a rats nest, because each little alley (branch) only has code for that team - well technically it has old stale code for all other teams from the moment the primary inventory-dev branch was taken.

Frankly I would say that following this anti-pattern isn't really a "mono-repo". It's just using one big repo as if it was 20 small ones - it doesn't really buy you anything, other than my first point about discoverability.

Perforce

Since the question mentioned perforce (note: my experience is rather dated):

Most places I worked, just created a single backend repo (depot?) then put all the code in there.

The critical question was where they created the branch point, if they put it at the top of the repository then effectively all teams are using the same set of branches.

If each team has their own branch point you effectively get the same problem as I outlined with Git, hunting an pecking for branches from each team.

Perforce has a "solution" for this which is client specs - you can define the client spec as a particular branch of each project.

Secondly perforce has a CL number which represents the last change committed ON ANY BRANCH so the combination of client spec and CL uniquely identifies all relevant code across all branches.

I air quoted "solution" because its really just moving the goal posts, you now have to manage the client-specs to ensure you are picking up the correct branches.

Same Problem / Same Solution

If you are going to use a mono-repo you need to have a REAL discussion about what branching and releasing looks like at your company.

Identify what branches you need and what they mean / will be used for at your company - ie: everyone is on the same page.

Once you have done that, you can use either source control product as mono-repo:

• In git - restrict your team to only using those branches **.
• In perforce - put the branch point at the top of your repository.

** - In git you can be a little be more flexible with early prototypes / feature branches, but you need to standardize, once the code is exposed to other teams.

Main-Only Mono-Repo

The comments discuss further restricting mono-repos to only using a main branch, which adds two further restrictions:

• All deployable artifacts (micros-services etc) are built/released off of main.
• All deployable artifacts should always be releasable - don't break the build.

In our case it didn't mean that all artifacts had to be released/deployed at the same time - although most of the time we did try to build/release all artifacts when there wasn't a specific reason not to.

It is still necessary to use the standard techniques for managing incompatible changes:

• Feature flagging.
• Versioning web services.
• Using versioning for items put on message queues.
• Writing to duplicate database tables/columns during transitional stages.
• ...

The most significant change was that it was no longer possible to make breaking CODE changes, simply by creating a new revision and then forcing all users of that code to update their code when they want to take the new version.

Instead the author of the breaking change was forced to do one of two things:

• Go through the entire code base fixing all places where the code was used - which was often the easiest solution when there were only a few locations.
• Support two versions of the API/function until such time as all teams have migrated to the new version.

There was a third option when the change was to a web-service - it was possible to have two versions (older and newer - both built off of main) of the same web-service deployed, each supporting a different version of the API. This came with its own issues, but there were times when it got us out of a jam.

• Ewan: Wow I hadn't even considered using branches as team/product separation, gona need to think about that one.
• Ewan: Its great to here from someone with real life experience of this. can you explain "if you need to build multiple products from different teams you need to check out different branches, build then switch to a different branch/tag" though? Is each team effectively ignoring the part of the repo they dont use and building a sub set from their feature branch?
• Ewan: have you seen an approach to monorepo that works? ie you can build the whole thing effeciently and have multiple teams work on different features at the same time? How did that work?
• DavidT: Only two projects I worked on got close to true mono-repos - both had a "no branch"/main only mandate. The perforce project leveraged the Perforce Pending CL’s feature, so that the changes had to pass automated testing, before it was submitted (visible to other devs). The git project exposed a single repo to me (as a dev), but there was a convoluted deployment process, which I didn’t have visibility into.
• DavidT: @Ewan - Added new section to address that TL;DR feature flagging is still useful but doesn't buy you anything extra - just because you are using a mono-repo.

Answer 2

I've been curious about this for a while, too. I think, ultimately, there is no single definition (as is the case for most software engineering buzzwords). I keep seeing the terms monorepo and monorepo-style development; this is more than "cram everything into one repository."

What is a monorepo?

Misconceptions about Monorepos: Monorepo != Monolith (Medium.com) has a concise list of attributes that a monorepo has, and it describes more than a version control strategy:

Monorepo-style development is a software development approach where:

• You develop multiple projects in the same repository.
• The projects can depend on each other, so they can share code.
• When you make a change, you do not rebuild or retest every project in the monorepo. Instead, you only rebuild and retest the projects that can be affected by your change.

You need to take all three aspects into consideration. Surely, version control plays a part in this, but the crucial aspects are that you have multiple projects that depend on each other, and this is a style of development; not just a version control practice.

The last point, that changes do not require a retest of every project, implies a change in team practices and how they use the CI/CD tools. You need some way to determine which tests should be re-run based on which projects changed.

With CI/CD tools being mentioned, it should be no surprise that a cursory search for monorepo comes up with a slew of companies hawking products aimed at monorepo-style development. I think "monorepo tools" is a red herring. Monorepos are buzzworthy, so of course companies will build and sell tools for it. I'm not saying CI/CD tools don't play a role in monorepos. It's just that companies are capitalizing on this to make you think you need a shiny new tool.

Of all the searching I did, I think the article on Medium.com has the best definition which is free of ulterior motives. Most other sources I've found are thinly veiled ads hawking a product, even if they have good information.

What problems do monorepos solve?

There is a bit of a story here. Semaphore.io has a good anecdote from Segment.com. Like many sources I found, Semaphore.io sells CI/CD software targeting monorepo development. Even so, I found it informative and a good illustration of how an organization arrives at this style of development.

Alex Noonan tells a tale about saying goodbye to multirepos. Segment.com, the company where she works, offers an event collection and forwarding service. Each of its customers needs to consume data in a special format. Thus, the engineering team initially decided to use a mix of microservices and multirepos.

The strategy worked well — as the customer base grew, they scaled up without problems. But, when the number of forwarding destinations passed the hundred mark, things started to break down. The administrative load of maintaining, testing, and deploying +140 repositories — each with hundreds of increasingly diverging dependencies — was too high.

"Eventually, the team found themselves unable to make headway, with three full-time engineers spending most of their time just keeping the system alive."

For Segment the remedy was consolidation. The team migrated all the services and dependencies into a single monorepo. While the transition was successful, it was very taxing as they had to reconcile shared libraries and test everything each time. Still, the end result was reduced complexity and increased maintainability.

"The proof was in the improved velocity. [...] We’ve made more improvements to our libraries in the past 6 months than in all of 2016."

Many years later, when a panel asked about her experience with microservices, Alex explained the reasons for moving to a monorepo:

"It didn’t turn out to be as much of an advantage as we thought it was going to be. Our primary motivation for breaking it out was that failing tests were impacting different things [..] Breaking them out into separate repos only made that worse because now you go in and touch something that hasn’t been touched in six months. Those tests are completely broken because you aren’t forced to spend time fixing that. One of the only cases where I’ve seen stuff successfully broken out into separate repos and not services is when we have a chunk of code that is shared among multiple services, and we want to make it a shared library. Other than that, I think we’ve found even when we’ve moved to microservices, we still prefer stuff in a mono repo."

^{Source What is monorepo? (and should you use it?), Semaphore.io}

Some key insights I got:

• Organizations transitioning from monolith architecture to micro services didn't just separate logic, they separated code bases to make independent teams.
• They found it very difficult to share code effectively without the overhead of a bunch of CI/CD pipelines - one for each micro service and shared library.
• Changes needed to be made across shared libraries and micro services in a coordinated fashion, but each team operating independent of the others made coordinated changes nigh impossible.
• Having all teams use a single repository facilitates this coordination, especially when changing shared libraries.
• Dependency management across the ecosystem became a soul-crushing grind with so many repositories. The transition to a monorepo was hard, but made dependency management a lot easier in the end.

When to use a monorepo?

Another company, Monorepo.tools, has some more good information. Again, they sell tools to facilitate monorepo development.

For the sake of this discussion, let's say the opposite of monorepo is a "polyrepo". A polyrepo is the current standard way of developing applications: a repo for each team, application, or project. And it's common that each repo has a single build artifact, and simple build pipeline.

The industry has moved to the polyrepo way of doing things for one big reason: team autonomy. Teams want to make their own decisions about what libraries they'll use, when they'll deploy their apps or libraries, and who can contribute to or use their code.

Those are all good things, so why should teams do anything differently? Because this autonomy is provided by isolation, and isolation harms collaboration.

^{Source: Monorepo Explained - But Why?}

They go on to list a number of drawbacks to polyrepos, which I think helps to define when a monorepo should be used:

• Code sharing is hard (think: custom built utility libraries)
• Significant code duplication (probably due to code sharing being harder).
• Costly cross-repo changes to shared libraries and consumers (again, probably due to code sharing being harder).
• Inconsistent tooling — I can see this being a problem if each team is given full autonomy to choose any and all tools, but full disclosure, this is where most sources slide into an advertisement for a product.

I don't think buying a new tool is a pre-requisite for monorepo-style development. Most robust CI/CD pipelines have all the tools you need (think: Jenkins, Azure DevOps, AWS, GitHub, etc). I'm not hawking those products either, I'm just remarking that if you already use a robust and fully-featured CI/CD pipeline, I doubt you need to buy additional tools.

So, when do you use a monorepo? When collaboration between teams suffers because you need to share code and make coordinated changes to dependencies. I don't think micro services are a prerequisite for using a monorepo, but it seems to have been the impetus for many organizations, and specifically around sharing code between micro services.

How to implement a monorepo?

Implementing a monorepo is not just a matter of:

cp projecta monorepo
cp projectb monorepo
cp liba monorepo
cd monorepo
git init .
git remote add origin ...
git add .
git commit -m 'Yay monorepos! Problem solved!'
git push origin HEAD

And now your problems go away. You still have multiple teams with different priorities and deadlines. Now they can coordinate code changes better. Teams become less isolated and less autonomous, but arguably, if you move to a monorepo, your teams are not as isolated and autonomous to begin with. Now your repository reflects these dependencies instead of pretending like they don't exist hidden behind a myriad disparate CI/CD pipelines and semantic versioning.

This implies a change in team dynamics requiring closer coordination. And it implies this closer coordination between teams solves more problems than it introduces.

• Ewan: Thanks for your answer. It does clear some stuff up. I can see that forcing teams to work on the same repo could well help them understand the bigger picture and work better
• Ewan: Hiwever.. as soneone who has always worked in what we can call a polyrepo approach. The problems alluded to woth the approach are not apparent to me. If you publish components that are consumed by a package manager. Where is the dependency hell?
• Ewan: Also, at the end you come to the conclusion that just by putting the projects into a single repo, that makes it a mono-repo. But, even if we accept that, is the approach feasible at scale without one of the build systems mentioned on the wikipedia page?
• Ewan: Doesnt a monorepo increase the dependency problem, in that if i make a breaking change i one component, im forced to update all consumers before I can complete my PR. Won't there be multiple inflight feature branches and merge hell?
• Caleth: @Ewan that isn't an increased dependency problem, it's the same dependency problem but more visible. If you make a breaking change in multi-repo, you might not know what you've broken, for whom. In both schemes you eventually need to coordinate between supplier and consumer to fix things up.

Answer 3

"monorepos" were really just attempts to solve problems with Perforce style source control where branching and merging have issues you don't see

I think this is wrong in general.

"just put all the code in one git repo and build it"

a simple behavioural change to a requirement has involved changing 10+ different git repositories ... Mono repos are preferable to this

I can sympathize with that view.

But if mono-repo just means "don't prematurely split a repo" it's not a revolutionary thing to blog about.

When has that ever stopped anyone from blogging about it anyways?? :-D

---

Because the to-mono-or-not-repo problem very much resonates with my work, I will try to give a perspective on what makes a monorepo, what distinguishing features it has and when one might prefer it:

Where I come from, the monorepo is partly a historical artifact, partly a result of fear of " ... a simple ... change ... involved changing 10+ ... repositories".

When the product was started, it was 1999 (or so about) and there was no source control. The product grew from 3 to 10 devs, and eventually SCCM was introduced. It had 100 thousands of lines of code, but it still was one product. Sure, it existed as a collection of programs and plugins, but they were all intricately linked, and surely released as a single product version, so all stayed in one repository (not git). Merging was never the problem with our system.

Over time, developers repeatedly raised concerns to separate out common logic into "proper" library repositories, just like they saw with the third party libs they used.

Some slowly-changing generic parts were factored out, but the vast amount of code, even common string processing utilities, still stayed in the one repo, because otherwise a dev touching anything there to facilitate a change in the final product would have to tackle 2 or 3 or 4 Jira tickets instead of one.

From this background, I can share the following:

• A monorepo should be a repository where code of different but tightly coupled programs and libraries are maintained together. (I mean, if it is just one program, then it's not a monorepo, it's just a repo.)
• Ideally all code is versioned at once. (not all parts neccessarily need to be released at once.)
• Testing procedures find all bugs introduced by a change in shared code across all usages.
• Refactoring across all sub-projects can be done by a single team without requiring intermediate compatibility releases of sub-components.
• Maintaining multiple major versions with bug fixes becomes arguably easier (see branching mentioned elsewhere).
  • When customers actively use (and get serviced) versions 1.x 2.x 4.x 7.x any fix in any version only requires a single repo to change, a single merge-chain to be applied to get the fix done in 4.3.67 back into main. With multiple repos this can quickly explode, if you need to merge multiple changes accross multiple dependent repos.

I think this can work well regardless of team size as long as you have one (or at least "one"-ish) product (or maybe call it "application suite") that is created through this repository.

It will start to fail when you have multiple products and/or product owners with different release and versioning requirements. Code parts shared among such separate entities should be factored out into "proper" libraries.

• Ewan: Does your "oneish product" rule mean it wont work with microservices? ie all the MSs in one repo
• Martin Ba: @Ewan - product, not program/app. If these microservices work together towards the same functional end-goal, I guess they can live happily ever after in the same repo. ;-)

Answer 4

TL;DR I'm not sure what a globally valid "current" definition is, but Google seems to share valuable perspectives of what a mono-repo can be.

---

I did not learn about the word monorepo from Facebook, but instead from Google.

I think whatever one of these has to say publicly about such a topic should have some weight on the question.

I think the first one was the 2014 talk by Titus Winters CppCon 2014: Titus Winters "The Philosophy of Google's C++ Code" although I'm not sure the word "monorepo" is actually used there. It's not in the slides.

Still, there is a page in the slides that gives a good datapoint for a monorepo:

... Context about Google

• 4K-ish C++ engineers
• Shared codebase
• Good indexer (Kythe)
• Good code review policies

... Most projects check into the same codebase. Most engineers have read access to most code. Most projects use the same infrastructure (libraries, build system, etc). ...

You can also find a paper where THE GOOGLE MONOREPO is explicitly mentioned:

... Google codebase in service of large-scale refactoring changes - broad but shallow changes that require many updates but little or no semantic change. ...

You can find further resources:

• https://qeunit.com/blog/how-google-does-monorepo/
• https://research.google/pubs/why-google-stores-billions-of-lines-of-code-in-a-single-repository/

---

So, to get back to the questions, I will give my own VERY succinct snippets:

• an explanation of the idea,

  • => All code is stored a single repository-like structure; developers can easily view and search through all code without having to check out a multitude of different sources.

• the problem it solves,

  • => refactoring at scale; quality control at scale; cross-project testing

• what situations it is applicable to?

  • => you have a coherent enough set of projects that have a consistent management umbrella to facilitate sharing code and tooling across many projects.
  • => you want to have immediate feedback when a change to one library breaks client code far removed from the triggering client code.

(These are my loose takeaways from following the Google side of this discussion - I'm sure to have mixed up some things in these short conclusions.)

• Ewan: thats an interesting paper. My take away is that he's proposing backwards compatible changes and restrictions on breaking changes to allow changes to gradually propagate across consuming projects. Avoiding merge issues and making updating to latest easy. Do you agree?
• Ewan: not sure how the 10k commit changes work? does he mean 10k files?
• Martin Ba: @Ewan - no, I think he actually means 10k commits. I'm no expert on the google strategy, I just followed it loosely (see my other answer for why) but I think Titus also has a talk about "live at HEAD" that further explains some of their strategies.
• Ewan: I see the other article lists less commits, but most are automated? so maybe thats it. I also note this quote : "Using Git would therefore amount to switching the monorepo to a multirepo model" where it seems to imply that monorepo is incompatiblew with git?
• Ewan: tbh though all these things while interesting make me question even more if this is what "the dev on the street" means when they say "monorepo" Google have a custom source control, a custom build system, a huge monolith codebase. and say monorepo is incompatible with git. The advantages are avoid dimond dependencies, avoid problems when you want to move a shared lib to its own repo.

Answer 5

Idea of monorepo is as strong as ever. The definition did not change.

They are uniquely suited for very large flat teams, where work is driven by user-facing feature, which is why large corporations use them.

Arguably hierarchical teams could also benefit from such, but Linux (where a large set of drivers is hosted in the main repo) process illustrates great overhead for those, negating the benefits.

Example

On request by @Ewan. Demonstrates a difference in handling a disruptive change in monorepo and multirepo.

• Imagine a large project like StackExchange
• Team size is in hundreds
• A policy requires each user-facing change to do an A/B test before being accepted.
• Two features are requested:
  • show a hat for user avatars
  • hide the downvote counter
• Each feature will take a few month to implement, verify, A/B test and accept
• Components to change:
  • backend
  • communication protocol (consider it to be a serialization library)
  • frontend

Monorepo flow

• Each task is assigned to a developer from UI team
• After a design stage and minimal approvals from the backend team, they begin development
• Modifications are done to all components simultaneously as needed
  • they change the communication protocol in a branch
  • request to do the backend work (in the branch) is sent to backend team
  • UI changes are done by assignee
  • when a CEO requests to add a tie to a hat, protocol is changed and a new request is send to backend team
• Each milestone is presented as a working prototype to testers
• Release candidates are A/B tested
• Hat feature is rejected by A/B test
• Hat branch is deleted
• Downvote feature is accepted and merged after a few months

Multirepo flow

• Each task is assigned to a developer from UI team
• After a design stage and extensive approvals from backend team, they begin development
• Modifications to a protocol are released as preview features. Hats are added, downvote count is zeroed-out by a feature flag. A minor version of protocol is released with the new feature.
• UI changes are done by assignee, in a UI branch
• request to do the backend work (in the branch) is sent to backend team
• backend team implements hats delivery under a feature flag and does nothing to downvote count (removal of downvotes would break current client) a minor version of backend is released
• when a CEO requests to add a tie
  • protocol is modified again, a minor version is released
  • a new request is send to backend team, a minor backend version is released
• Each milestone is presented as a working prototype to testers.
  • Testers are instructed to use feature flags
  • Milestones are harder to do, because changes to protocol need to be coordinated
• Release candidates are A/B tested with feature flags
• Hats feature is rejected by A/B test
• Hats branch is deleted from UI
• Hats are reverted from backend, a major version of backend is released
  • backend team has to accomodate the breaking change when working on downvote feature
• Hats are reverted from protocol, a major version of protocol is released
  • the whole team has to accomodate the breaking change when working on downvote feature
• Feature flags are removed
• Downvote feature is merged, a minor version of UI is released

This example demonstrates, that a parallel development of multiple features is much more complex in a multirepo paradigm, because each change is hindered by release management and coordination, feature flags are used to control changes unbound from timeline.

• Ewan: why are they suited to large teams? surely as the code size increases and the frequency of commits increase the frequency of merge conflicts and hence build failures increases?
• Basilevs: @Ewan multirepo is worse in this regard, because commits are not synchronized. What would be a bad merge in a monorepo, detected by PR verification and rejected becomes an unstable build in multirepo, as dependencies are either updated of fixed to incompatible versions.
• Ewan: I don't understand the second case. If I update a library in muti-repo, a new version is published. I then switch to the consuming repo, and make the update to use the new lib + required code changes. Each PR is a small change with no conflicts. What is an "unstable build" in this case?
• Basilevs: Multirepo allows a subcomponent to force decisions on consumers, while monorepo prioritizes working state of all final products. Obviously, any business would like their value-adding products to work.
• Ewan: Your example totally works with multirepo though? B can release 1.1 with the required change and A can upgrade? or You could decide, lets upgrade A to use B v2. In the monorepo you can never write B v2