Guix Android

This is a Guix channel that provides additional packages for Android development. This channel is about packaging software that is not yet in Guix, but that might get there once it's ready.

You can install and use this channel, but consider it to be very beta, many things might not work well. The main motivation for this channel is to provide the Replicant project a way to build their system and applications using an FSDG approved distribution. See this page for more information on the relation between Replicant and Guix.

What Works

Currently, building C and Java components seem to work well enough. We have the following components available:

• adb, the Android Debugging Bridge;
• fastboot, to flash ROMs on your devices;
• simg2img and img2simg for manipulating sparse images.
• aapt, etc1tool, hprof-conv, split-select and zipalign which are additional tools for the SDK.

Installing

This is a Guix channel. You will first need to install Guix itself. Then, simply create a new ~/.config/guix/channels.scm file with this content, or update the file with the additional channel if it already exists:

(cons* (channel
 (name 'guix-android)
 (url "https://framagit.org/tyreunom/guix-android.git")
 (introduction
 (make-channel-introduction
 "d031d039b1e5473b030fa0f272f693b469d0ac0e"
 (openpgp-fingerprint
 "1EFB 0909 1F17 D28C CBF9 B13A 53D4 57B2 D636 EE82"))))
 %default-channels)

Then run guix pull to pull the new channel.

Important checks

Make sure your guix environment is set up properly. You need to have ~/.config/guix/current as the first item in your $PATH or you're going to run into troubles. Additionally, after running guix pull, make sure you run hash guix in any open terminal to make sure bash's cache is cleared of the old guix binary location.

Contributing

Contributions are welcome! You can create a merge request, or send me a patch by email, whichever you are more familiar with.

If you have a history of making quality contributions to GNU Guix and would like commit access, just ask! Non trivial changes should still go through a simple Merge Request and code review process, but this channel needs your help to take over the world!

Do not hesitate to get in touch if you have an idea or want to help in any way!

Plan for Integrating Android and Guix

Android is composed of many repositories. Official instructions are to download all of them and select a target you want to build. Many products can be built from these repositories:

• Host tools, that interact with Android from your non-Android machine. These are for instance adb, fastboot, etc...
• The SDK (Software Development Kit) that allows you to build android applications from your machine. This is composed of various components: the Android library, host tools to build an apk, ...
• The NDK (Native Development Kit) that allows you to build native code embedded in applications from your non-Android machine.
• Android itself and its variants, cross-built from your machine.

The goal of this repository is to package all of them, starting from the simpler targets that are the host tools.

Fetching Sources

Fetching these sources is not easy, and Google provides a tool called repo that reads a manifest file and fetches the declared repositories in a single directory. In this work, we will build the various packages independently, and therefore we will not create a huge build directory, but rather build from various git repositories.

The goal of the (android import repo) module is to implement an importer that lets you download the whole set of sources, so you can see and analyze them. It does not currently implement the importing feature, but it can read a manifest and download the declared repositories.

You can use it from the guix repl:

,use (android import repo)
(let ((manifest (get-manifest)))
 (fetch-manifest-repositories manifest))

• Scheme Procedure get-manifest [#:url %default-manifest-url] [#:version %latest-android-12] [#:cache-directory (%repository-cache-directory)]

  Download the given manifest for the given version, in the given cache directory. By default, it will download the manifest for the latest stable version of AOSP in the default cache directory (~/.cache/guix/checkouts). This procedure returns a manifest object.

• Scheme Procedure fetch-manifest-repositories manifest [#:cache-directory (%repository-cache-directory)]

  Download the various repositories for the given manifest object. By default, it downloads in the default cache directory, but since the result is pretty big (more that 25 GiB), you may want to specify another directory in another file-system. Passing the same manifest again will complete an incomplete download or ensure everything is downloaded. Passing a different manifest will create a new directory where the sources will be fetched again. Every clone is shallow, but still take a lot of space.

Importing Packages

The (android import repo) module also provides an importer. This importer does not use the shared importing utilities, because it requires a big state to be carried over between package imports. This structure is quite long to generated, so we use a single procedure for recursive imports, so we only need to generate it once.

You can import packages from a REPL and generate a package and source file, similar to what we currently have in (android packages android-sources) and (android packages android-tools):

,use (android import repo)
(define manifest (get-manifest))
;; Fetch repositories that form the android distribution. This is very big
;; (> 28GB) and will take some time. If it fails because of a network issue,
;; restart that command again until everything is downloaded.
;;
;; You can also add an optional destination directory where the sources are
;; to be downloaded, e.g. on a different file-system if you lack space on
;; the file system your home directory is on.
(define root (fetch-manifest-repositories manifest))
;(define root (fetch-manifest-repositories manifest #:cache-directory "/mnt/hdd/android-sources"))
;; Finds all Android.bp files from which to import, takes ~10s on warm cache,
;; significantly longer on cold cache and spiny disks.
(define bp-files (get-all-bp-files root))
;; Creates the shared structure: a map of modules and variables to the
;; files that define them, so we can find them, without actually parsing
;; everything. Will take ~5mins on an HDD, could be faster on SSD.
(define bp-maps (get-bp-maps bp-files))
;; Run the import. The last two arguments are the names of files that will
;; be generated.
(import-recursively manifest root bp-maps '("adb" "fastboot")
 "sources.scm" "packages.scm")

Building Modules

Android uses the Soong build system. This system reads Android.bp files in the various repositories and run the declared build commands.

Soong is pretty hard to package and assumes in many places that it is run from a single directory that contains all the sources. Furthermore, it relies on pre-built binaries and does not allow us to use system packages, for ``reproducibility'' purposes, ignoring bootstrappability.

This channel re-implements soong and its build system in the soong-build-system, which does not use soong at all. Inspired by the work pioneered in soongnix, we propose a build system that:

• Parses Android.bp files to modules
• Checks the module type and options, and generates a Makefile
• Run the Makefile to build and install the module

Following that plan, we will end up with thousands of individual modules. The role of the importer will be very important, as it will be the only way for us to properly package and maintain such a big collection.

The build system is currently implemented in this way:

(android build-system soong) is the build system itself that must be imported for packages to use the soong-build-system. One key feature is that the set of implicit inputs depend on the type of module we build. For example, a cc module will use clang and llvm, while a genrule module does not have any additional implicit inputs.

(android build blueprint) implements a parser for Android.bp files. This will be used later in the importer, and is currently used in the build system to learn the various options for a package. It contains the following variables:

• Scheme Variable %latest-android-10

  The latest version number in the android 10 series.

• Scheme Variable %latest-android-11

  The latest version number in the android 11 series.

• Scheme Variable %latest-android-12

  The latest version number in the android 12 series.

This module also defines the following low-level procedures.

• Scheme Procedure uniq lst [#:equal? equal?]

  Returns the input list with only uniq elements, that is the elements that are not equal? to another.

• Scheme Procedure android-name->guix-name name

  Returns a proper guix name given a blueprint module name.

This module also defines the blueprint-related datatypes.

• Datatype blueprint

  The parsed content of a blueprint file.

  • vars

    The list of variables in the file, as an association list relating the name of the variable to its content.

  • modules

    The list of modules in the file, each represented by a blueprint-module object.

• Datatype blueprint-module

  The content of a single module in a blueprint file. For instance, the following blueprint contains two modules, foo and bar:

  cc_library {
   name: "foo"
   srcs: ["foo.c"]
  }
  cc_binary {
   name: "bar"
   srcs: ["bar.cc"]
   shared_libs: ["foo"]
  }
  

  • type

    The type of module, such as "cc_library" or "cc_binary" in the previous example.

  • name

    The name of the module, such as "foo" or "bar" in the previous example.

  • keys

    The set of configuration keys, as an association list. For instance, the keys of bar is ((srcs . ("bar.cc")) (shared-libs . ("foo"))). Note that the keys are normalized: underscores (_) are replaced by hyphens (-).

  • defaults

    The list of default modules that still need to be considered before the keys are complete.

  • file

    The name of the file this module comes from.

This module further defines the following high-level procedures:

• Scheme Syntax define-module-record-type name make predicate field-spec ...

  Defines a record type called name, with the constructor make and predicate predicate. field-spec is a set of specifications, each of them containing a field name, an accessor name and a default value. Eg:

  (define-module-record-type foo make-foo foo?
   (a foo-a (default 1))
   (b foo-b (default "a")))
  

• Scheme Procedure get-module bp-files module-type module-name

  Return the module object for the module named module-name of type module-type (or any type if it is #f) in bp-files, a list of file names.

• Scheme Procedure generate-bp-file file module

  Create a file that contains the blueprint representation (Android.bp format) of the module object.

It also has the following low-level procedures:

• Scheme Procedure parse-blueprint-file file

  Return a parse tree for the given file.

• Scheme Procedure parse-tree->blueprint tree file

  Transforms a parse tree to a blueprint object, without resolving variables nor default modules. file is the filename from which the tree is read.

(android build soong-build-system) implements the phases for the soong build system. Its configure phase creates a Makefile by generating various rules depending on the type of module. It also installs share/android/Android.bp, a blueprint file that contains the parsed values for the module being built.

(android build soong) is a support library used by the soong build system. It implements the following procedure:

• Scheme Procedure expand-srcs srcs inputs

  Given the set of sources srcs and the association list of inputs inputs, return the expanded set of sources. This procedures does two things: it expands any wildcards and replaces package references (sources that start with ":") by the package's content.

(android build soong makefile) contains the necessary code for generating Makefiles. It implements the following data types:

• Datatype makefile-rule

  This datatype represents a rule in a Makefile. It contains the following fields:

  • out

    The list of files this target creates.

  • deps

    The list of files required by this target.

  • cmd

    The command to run to generate the output files.

  • source?

    Whether this target generates sources. Generated sources are always built first. None generated sources depend on generated sources targets.

  • dependencies

    The list of external dependencies required by this rule. For instance, if the rule runs the compiler with a -lfoo flag, it is likely that it requires the "libfoo" package to be present.

    This is used only in the importer.

• Datatype makefile

  This datatype represents the content of a Makefile. It contains the following fields:

  • variables

    A set of variables, defined as an association list.

  • build-rules

    A set of build rules, each of them being a makefile-rule object.

  • install-rules

    A set of installation rules. Each installation rule is a list of three elements: the type of installation rule, the file or directory to install, and the target directory, relative to the output directory in the store. Installation types include:

    • file, for installing a single non-executable file.
    • executable, for installing a single executable file.
    • dir, for installing the contents of an entire directory.

It also implements these procedures:

• flatten lst

  Return the list passed as an argument, after flattening it. E.g. (flatten '(1 (2 3 (4 (5))) 6)) is (1 2 3 4 5 6).

• generate-soong-makefile makefile [#:name "Makefile"]

  Creates a Makefile from the given makefile object. The file is written and the return value is unspecified.

(android build soong cc) and other similar modules implement various rules for generating a Makefile. They all specify a configuration type using define-module-record-type, and a procedure to generate a makefile object from the given configuration.

Cross-Building

Cross building is supported by the soong build system, for any architecture supported by Guix (glibc-based). Future plans include adding support for *-android architecture, but that requires using bionic libc instead of glibc, which is built using the soong-build-system.

The plan is to import bionic into guix, tweak the cross toolchain for the android architectures and use that for cross-building Android.

Unfortunately, libcxx and libcxxabi (see (android packages clang)) do not build with a cross-compiler at the moment. Help wanted!