How to organize my Arduino projects' structure for easy source control?

Question

It's been a long time I've been looking for a good answer to this question.

Typically, any Arduino project but the simplest one will include:

• The main source code file MyProject.ino
• Libraries specific to the project (MyProjectLibrary1.h, MyProjectLibrary1.cpp...)
• Third-party libraries (generally free open source, added manually to Arduino libraries directory)
• Schematics, PCB diagrams
• Documentation
• ...

All this makes it hard to keep the whole code and doc of one project under Source Code Management (e.g. on Subversion, Git or GitHub).

Managing source control of your project means managing the version of all files used by the project including 3rd-party libraries.

Now for a single project, I need to define a directory structure that:

• Includes all project files as described above
• I can entirely commit to a Source Code Management tool (including 3rd-party dependencies)
• I can checkout anywhere on my hard drive and build the project from there (does it have to be a single location as imposed by Arduino IDE)
• I can zip into a self-contained archive that I can send to a friend for him to build as easily as possible (no extra manual download)

What I find particularly tricky with Arduino projects is the management of external libraries dependencies. Java projects developers have maven repositories for that and that helps a lot in managing all external deps. But we don't have an equivalent system for Arduino libraries.

I would be interested to know how other Arduino project makers deal with these aspects in their own projects.

Also note that I am open to changing my development process, including my IDE (currently I use Eclipse with the Arduino plugin most of the time, and then I ensure my projects can also work directly with the Arduino IDE).

• Cybergibbons: I've also been struggling with this. I have two projects which need different versions of an external library, and currently they are outside of version control.
• kaiser: Please note for further research that those are Package Managers. JavaScript got Node.js/npm and bower, PHP got PEAR and Composer, etc.

Answer 1

My way of organizing an arduino project is quite simple, all my project are git repositories so that there is at least the following:

• project/
  • project.ino
  • Makefile

I have a preference using my favorite editor and a Makefile which I have crafted to work against most use cases (and I even improved on that one that I'm going to share soon).

For the libraries, I prefer to keep them as their own repositories and use git submodule to include them to the project. As many libraries written by the community are shared as git repositories, that's a good generic solution. Then, within the Makefile, I just have to add the libraries path I want to include in the LOCALLIBS variable.

Though, for some projects, it makes sense to encapsulate the libraries into a hardware abstraction layer library crafted for the project, then I prefer to use a path such as:

• project
  • project.ino
  • Makefile
  • project_hal_lib
    • library1
    • library2
    • library3
    • ...

Though, with arduino 1.5.x a new way to specify libraries is offered, that will offer a way to create and build arduino projects the same way we already do with pipy and virtualenv in python, i.e. you define the set of libraries you need and they get downloaded.

• asheeshr: I was working on a similar answer. You beat me to it!
• jfpoilpret: +1 Thanks! This way looks quite interesting. I'll have to give it a try this week (I need to check out how to setup the Makefile stuff first, though).
• jfpoilpret: @AsheeshR if your answer was similar, that means it still has some differences, right? I'd be interested to know about these!
• zmo: actually the main changes to come with next version of my Makefile will be the ability to either flash using a programmer or upload using the bootloader. As well as handling merging of bootloader with firmware. I also wrote an in-makefile fuse setter.
• jfpoilpret: @zmo bounty deserved, though your Makefile solution cannot work in my situation (using Windows but I did not specify that point). But I am convinced that using one of the existing makefile solutions available is the way to go. Once I have found one tha tworks for me I'll post my answer here.

Answer 2

The simplest way to do this is to copy the header and code files of the library into your source directory and include them.

myproject/
 myproject.ino
 somelib.h
 somelib.cpp

In your code, you can do include "somelib.h"

The down side to this is that the libraries must be in the same folder, not sub folders, so it makes your directory look messy.

---

Regarding the directory structure of my entire project, including schematics and documentation, mine usually looks like this:

myproject/
 schematics/ - eagle files or whatever you may have
 docs/ - include relevant datasheets here
 test/ - any test cases or other code to test parts of the system.
 myproject/ - since Arduino code must be in a directory of the same name
 myproject.ino
 ...

• jfpoilpret: Another downside is that I will have to copy the same libraries on many projects. Also, it is not clear to me if you put only YOUR libraries in there or also 3rd-party libraries?
• sachleen: First point: That's not really a down side, that's just a side effect of keeping libraries and project source together as you wanted with version control. What if another project needs an updated version of the library? What if you modified it? Second point: both will work.
• sachleen: I do not. The Arduino IDE is quite limited in many ways. You may want to look into a better environment to work in that has better support for this. People have made custom make files that let you import libraries from other sources as well.
• asheeshr: This is not a good way to organize projects from the point of view of software licenses. If you are including third-party libraries in your project, which may have different licenses, then you may be violating them as soon as you start sharing the project file. Different open source licenses are generally not compatible with each other.
• sachleen: @AsheeshR having all your files in one directory so the arduino IDE doesn't complain is not a good way to organize projects at all. It's just a way. Feel free to propose a better solution. I don't know of one that still allows you to use the Arduino software.

Answer 3

Git submodules are extremely powerful when it comes to organizing multiple nested repositories. Handling multiple libraries from different sources, and even handling parts of your own project which may be stored at different sources becomes easy with git submodules.

Directory Structure

A way to organize your projects would be:

• projectA - Parent Directory

  • projectA - Source code directory containing Arduino code

    1. projectA.ino
    2. header.h
    3. implementation.cpp

  • docs - Your main documentation directory

  • schematics - these may be maintained separately on a separate Git repo or part of the same repo

  • libs - This will contain your third party libraries.

    1. libA - These may be maintained as third party repositories
    2. libC - ...

  • license

  • README

  • Makefile - Necessary to handle dependencies across directories

Workflow

You would follow your normal cycle of make changes, add and commit as far as the main repository is concerned. Things get interesting with the sub-repositories.

You have the option of adding a repository into the parent directory of your main repository. This means that any part of you directory structure, i.e. docs, schematics, etc. can be maintained as a separate repository and continuously updated from.

You can do this using the git submodule add <repo.git> command. To keep it up to date, you can use git submodule update <path>.

When it comes to maintaining multiple third party libraries within your repository such that each can be version controlled in itself or each can be kept up to date if need be, git submodule again saves your day!

To add a third party repo to libs, use the command git submodule add <lib1.git> libs/lib1. Then, to maintain the library at a fixed point in the release cycle, checkout the library and make a commit. To keep the library up to date, use the command git submodule update <path>.

Now, you can maintain multiple repositories within a main repository as well as multiple third party libraries in their independent stages of release.

Versus Single Directory Approach

While the single directory approach is the simplest, it is not possible to version control parts of a directory without a lot of pain. Hence, the simple approach fails to accomodate different repositories with varying states in the project.

This approach allows maintaining multiple repositories but brings in the need for a Makefile to handle the compilation and linking process.

Depending on the complexity of your project, the optimal approach can be selected.

• kaiser: +1, but just as a sidenote: Git Submodules are pretty unstable and likely loose track. It makes no difference if you use a single directory or multiples (like vendor, node_modules, etc.). Git references them and keeps track of it.
• asheeshr: "It makes no difference if you use a single directory or multiples (like vendor, node_modules, etc.)." I didn't understand this part. Could you elaborate?

Answer 4

Here is the way I finally decided to follow for my projects.

Arduino-CMake

The first important decision I made was the choice of a build tool that could work for my environment (Windows) but was not limited to it (I want my projects easily reusable by other people).

I have tested various open source Arduino make tools:

• Guyzmo Makefile (suggested by @zmo answer): this is just a standard Makefile handcrafted for Arduino builds; this is a Unix Makefile but there is a good port of Unix make for Windows; still, unfortunately this Makefile works only for Unix, of course it could be adapted for Windows, but I wanted a tool that works "out of the box".
• Arduino-Makefile (suggested by @adnues answer): this a more advanced project, based on Unix Makefile, that aims to be easily reusable by all Arduino projects; it is documented as working on Mac, Linux and Windows, but Windows support proved wrong in my first experiments (many dependencies on Unix shell).
• Graduino (not suggested by any answer): this build tool is based on well-known gradle build tool from the groovy world; the tool seems quite well done but requires some (little) groovy/gradle knowlegde, and has only little documentation; I decided not to go with it due to the burden of installing groovy and gradle just for it (I'd like to avoid too many pre-requisites to people who want to build my projects on their environments).
• Arduino-CMake (not suggested by any answer): this seems the best of all, it has a long history, has many supporters and maintainers, is very well documented, comes with simple examples and also has a few good tutorial blog posts on the Web, e.g. here and there. It is based on CMake, a "Cross-Platform Make".

I have also found ArduinoDevel , another Arduino build tool -which I have not experimented- that can generate Unix Makefiles or ant build.xml files; that one seemed interesting but a bit limited in terms of functionality.

Finally I decided to go with Arduino-CMake:

• it was easy to setup: just install CMake on your machine and copy Arduino-CMake in some directory that is easily accessible (through relative paths) from your projects directories.
• the examples worked out of the box for me (just followed the comments in CMakeLists.txt configuration file to adapt properties that needed to for my environment, e.g. Arduino type, serial port)
• you can organize your projects any way you want
• it can generate configuration files for various build tools out there (I have only tested Unix Makefiles though), including Eclipse projects.

• in the generated make, several targets get created to support:

  • libraries build
  • programs build
  • program upload to boards
  • serial monitor launch
  • and a few others I have not tested yet

Project Structure

Since Arduono-CMake does not impose any directory structure for your project, you can choose the one that fits you best.

Here is what I've done personnaly (that still requires further refinement, but I'm happy with it now):

enter image description here

I have decided to put all my projects under a common arduino-stuff directory (which I commit to github as a whole, I know I could use git submodules for a better organisation on github, but had no time to check that yet).

arduino-stuff has the following content:

• build: that's a directory where cmake and make will generate all their stuff (makefiles, cache, object files...); this one does ot get committed to github
• cmake: that one is just a copy (unmodified) of Arduino-CMake cmake directory. This one gets on github so that it's easier for someone who wants to build my projects
• CMakeLists.txt: that's the "global" CMake configuration that declares all defaults for my environment (board, serial port) and the list of build target subdirectories
• TaskManager: this is my first project based on Arduino-CMake, this one is a library with examples; this idrectory also contains a CMakeLists.txt that states the targets for the project

Points to improve

The current solution is not perfect though. Among improvements I see (that's rather for Arduino-CMake project to include these improvements if they see fit):

• Feature to copy a library directory from the current project to the Arduino libraries directory
• Feature to upload a library to github
• Feature to download a library from github

• ohhorob: Have you tried PlatformIO yet? It might not have been around when you were asking this question.. platformio.org
• Edgar Bonet: It looks like the project died and resurrected a few times: arduino-cmake → forked arduino-cmake → Arduino-CMake-NG → Arduino-CMake-Toolchain.

Answer 5

MyProject
|_MyProject
 |_MyProject.ino
 |_data
 | |_documentation 
 | |_PCB
 | |_schematics
 |_src
 |_MyProjectLibrary1
 |_ThirdPartyLibrary

MyProject folder (repository root)

The reason I suggest the seemingly redundant MyProject root folder is that you mentioned using GitHub. When you download (rather than clone) the contents of a GitHub repository the branch or tag name is appended to the repository name (e.g. MyProject-master). The Arduino IDE requires that the sketch folder name match the sketch file name. If you open an .ino file that is in a folder that does not match the sketch name the Arduino IDE prompts you create an appropriately named sketch folder and move the sketch to that folder. In addition to this not being a very good initial experience for the user, the greater problem is that the Arduino IDE may not copy all the other associated files to the newly created folder, which could cause the program to no longer compile. By putting the sketch in a subfolder you avoid GitHub altering the name of the sketch folder.

If the GitHub file name thing is not an issue for you then the redundant root folder is not necessary.

data folder

I recommend using the data subfolder for your non-code files because the Arduino IDE has a special treatment of subfolders of that name. They are copied to the new location when you do a File> Save As.... Subfolders of any other name are not.

src folder

The src subfolder has the special property of allowing recursive compilation. This means that you can leave the libraries in that folder and include them from your sketch like this:

#include "src/MyProjectLibrary1/MyProjectLibrary1.h"
#include "src/ThirdPartyLibrary/ThirdPartyLibrary.h"

The Arduino 1.5 Library format folder structure is also supported, you only need to adjust your #include statements accordingly.

Note that only Arduino IDE 1.6.10 (arduino-builder 1.3.19) and newer support recursive sketch compilation.

Unfortunately some libraries use the incorrect #include syntax for local file includes (e.g. #include <ThirdPartyLibrary.h> instead of #include "ThirdPartyLibrary.h"). This still works when the library is installed to one of the Arduino libraries folders but does not work when the library is bundled with the sketch. So some libraries may require minor edits to use this way.

I greatly prefer this to the alternative of dumping all the library files in the root of the sketch folder because that is messy and every library file will be shown in the Arduino IDE as tabs when you open the sketch (Of course any source files you do want to be editable from the Arduino IDE should be placed in the sketch root folder).

Being able to use bundled libraries in place is also in line with another of your goals:

send to a friend for him to build as easily as possible

Removing the requirement to manually install libraries makes the project much easier to use.

This will also avoid any chances of conflict with other versions of library files of the same name that may be previously installed.

Answer 6

You could use the makefile https://github.com/sudar/Arduino-Makefile for compiling Arduino codes. You don't necessarily need the IDE.

• jfpoilpret: I have tried but unfortunately it will work only on Unix machines and I need Windows support. Currently I am evaluating another project based on CMake but I'm not done with it yet. I'll post an answer when I have decided on a tool.

Answer 7

Probably really late to the game but it's a popular enough question to answer using slightly different methods than those posted already.

If you need to maintain compatibility with Arduino IDE directly you could use something like the one I outlined here:

https://gitlab.com/mikealger/ExampleArduinoProjectStructure/tree/master/ExampleSketchBook

I based most of this off the notes of Arduino -- Project structure and build process, and some tips I have picked up over the years.

I really don't know why this is so hard to find via Arduino pages directly, it seems silly coming from a semi professional background that the build process is so obtuse.

best of luck out there

----edited link to the project structure notes i used as they have been erased they are now here

• Greenonline: The gitlab link appears to be broken
• Mike Alger: odd does it work with out the the direct link ? i.e. gitlab.com/mikealger/ExampleArduinoProjectStructure
• Greenonline: Actually both links work in Firefox, but neither work in my outdated Chrome version 49.0.2623.112 (64-bit). Nothing to worry about, I guess. :-)
• Mike Alger: ok grahamwideman's page is gone now but the wayback machine has a copy at web.archive.org/web/20161106122158/https://…