Where, in a repository, should I put source code for tools used in the build only?

Question

Suppose I have a repository for an application app_a. To build app_a, one needs to compile some sources (e.g. file1 and file2, never mind the file suffixes), but - it is also necessary to:

1. Apply some utility executable util_b to file1.in to get file1.
2. Apply some utility executable util_c to file2.o, the compiled object of file2 , to get file2_extra.o, which is used in linking app_a

If util_b and util_c's sources are in another repository - then no problem, we just have to depend on them being available on the path, and not our problem.

My question is: Suppose that util_b and util_c are bespoke (i.e. not generic, not used elsewhere, custom), and their source are part of my repository. Where, in the repository, should I put their sources? And - should I treat them as just additional source files on which the executable indirectly depends, or should they be separated more strongly?

Notes:

• I realize the answer for util_a and util_b might be different
• I am mostly interested in an answer for a C++ or C repository, with an idiomatic structure, e.g. as described in the Pitchfork repo or in P1204R0 Canonical project structure paper.
• If it matters, assume the application build is managed via some build system generator, e.g. CMake, meson, autotools etc.

• Doc Brown: This question looks similar to this one Organizing Ad-Hoc Tooling in a Multi-Repo Structure. Do you think the accepted answer fits to your case as well?
• Ewan: it doesn't make sense to me that you would put the source code for these build apps in the same repository as a completely different application? Surely the answer is "In a different repo"
• DavidT: @Ewan some companies have a "mono-repo" standard/process, however I think that Doc Browns linked question still covers the concept (with appropriate adaptation).
• Ewan: i'm not sure "mono-repos" exist outside that one facebook post which was custom source conrtol
• Doc Brown: @Ewan: if it helps yout to extend your point of view, our team uses a mono-repo approach for several hundred inhouse-tools for almost two decades, and we are still satisfied with it. And no, we are not working for Meta inc.

Answer 1

I will consider two situations

The tool is generic tool

With a generic tool, I mean a tool that can be used outside the context of app_a.

In this case, there is no intrinsic reason to keep the source of the tool with the source of app_a. It might even become confusing when the tool also starts to be used for app_b.

With such a tool, I would recommend to put it in a repository of its own (or a repository of tools) that has its own release cycle. From app_a you should treat the tool the same way as an externally provided tool.

The tool is specific to app_a

If the existence of the tool only makes sense in the context of app_a, then it can make sense to have the sources also in the app_a repository.

Within the Pitchfork project structure, the tools/ folder would be the correct place for it. Although it is not explicitly described, for a tool that needs to be compiled first, I would create a tools/<tool name> folder and re-create the folder structure under it as-if it were a new root folder. This way, the sources for app_a and the tool will not be mixed, but everything will still be in a familiar folder structure.

• einpoklum: I did say the two tools are bespoke, i.e. that your second paragraph applies; perhaps I was not explicit enough when saying that; will edit the question to better clarify this. But - the first part of your answer is also relevant, even if not to my case.
• einpoklum: Also, you describe what you would do; but - do you know this to happen in practice?
• Bart van Ingen Schenau: @einpoklum, we also have some custom tools that we use, but those tools are used for multiple products. They are bespoke to the company, but not to a product and that is when the first case applies. And I would apply it even if there is a possibility of the tool being useful for multiple products.
• Bart van Ingen Schenau: In my workplace, it is rare to have product-level bespoke tools and the few that we have tend to be python or matlab scripts, so there is no (explicit) compilation step involved. So, I can't definitively say that the second option describes how things are done, but that is how I would do it if I faced that situation.

Answer 2

Given the referenced specifications for project structure it should be quite easy to find the answer and im sure you have already looked

Pitchfork

: [[#tld.tools|tools/]]
:: Directory containing development utilities, such as build and refactoring
 scripts

or

: [[#tld.external|external/]]
:: Directory for packages/projects to be used by the project, but not edited as
 part of the project.

P1204R0

Unclear. Really just talks about single project libraries

Out of the two, pitchforks /external seems to be the best match

The external/ directory is reserved for embedding of external projects. Each embedded project should occupy a single subdirectory of external/.

external/ should not contain files other than those required by tooling.

This directory may be automatically populated, either partially or completely, by tools (eg. git submodules) as part of a build process. In this case, projects must declare the auto-populated subdirectories as ignored by relevant source control systems.

Subdirectories of external/ should not be modified as part of regular project development. Subdirectories should remain as close to their upstream source as possible

It also points to the possibility of populating it via git submodules, or pulling from a source. Which seems the more sensible solution, why wouldn't these utils be used by more than one project?

• einpoklum: "Directory containing development utilities, such as build and refactoring scripts" <- but I'm talking about the sources for those scripts.
• Ewan: yes, that's why I think external is a better fit. You could make an argument for tools on the basis of the use of the script though
• Caleth: Isn't a script it's own source? Either way, I still think tools because it is part of how the project is built. Unless you are talking about something like building Qt from source, that fits external
• Ewan: you can make an argument for either. My recommendation is external

Answer 3

I'm not a C/C++ dev, so I cannot account for any prevailing conventions for those languages specifically.

Generally, I tend to structure my repository with several top-level folders:

root
 /apim
 /docs
 /pipelines
 /src
 /test
 MySolution.sln

This is a C# example for an Azure-oriented app, as some of the names reveal.

This keeps the different parts of a service's lifecycle neatly separated.
Note that I'm not opposed to having tests be a subfolder within the source code, I think that's subjective and the distinction is not particularly relevant for the current question anyway.

/pipelines is really just the build folder (which in our case comes in the form of ADO pipelines. But it does have a purpose that's very relevant for your question: storing all the resources that are only used for the build process itself, not for local development or deployed runtime.

For your scenario, I would rename that /pipelines folder to /build, and store all build-related resources in there.

---

In case the question comes up for other readers, when dealing with a monorepo the structure is basically the same except that the root contains service folders, and the service folders look like the root in the above example.

root
 /ServiceA
 /build <--- build resources specific to service A
 /docs
 /src
 /test
 ServiceA.sln
 /ServiceB
 /build <--- build resources specific to service B
 /docs
 /src
 /test
 ServiceB.sln

If, in this case, the build tools are meant to be reused across services, I would hoist the /build folder up to the top level. Given that monorepos tend to have a lot of services in them, a prefixed /_build seems appropriate so it does not get lost in the list of service folders.

root
 /_build <--- build resources not specific to a service
 /ServiceA <--- same content as previous example
 /ServiceB <--- same content as previous example

• einpoklum: Thanks for the effort, but - this answer is less relevant given C and C++ repository structuring conventions.

Answer 4

It depends on the capabilities of your build system. The best case is that everything is in one repository, you check it out, press "build", and some time later your app is built and you can run it.

Can your build system achieve this? Can it figure out which tools are needed and build them automatically? If yes, put everything into one repository. Do you have to build your tools manually? It might be better to put them into a separate repository.

• einpoklum: Of course it can, the question is only about directory organization. Suggest you delete this, it's likely to get downvotes :-(