Contributor Covenant

iod-stats-builder

The scripts in this repository allow you to turn your web server logs into IzzyOnDroid download statistics, compatible with iod-stats-collector.

The conversion process is done in two steps.

First, the web server's log files are read and converted into a JSON file with download statistics. This JSON file is formatted as follows:

{
 "2024年12月08日": {
 "InfinityLoop1309.NewPipeEnhanced": {
 "Droid-ify": 274,
 "F-Droid": 52,
 "F-Droid Classic": 1,
 "Neo Store": 5,
 "_unknown": 167,
 "_total": 499
 },
 "com.akylas.weather": {
 "Droid-ify": 62,
 "F-Droid": 25,
 "F-Droid Classic": 0,
 "Neo Store": 2,
 "_unknown": 39,
 "_total": 128
 },
 "com.machiav3lli.fdroid": {
 "Droid-ify": 0,
 "F-Droid": 3,
 "F-Droid Classic": 0,
 "Neo Store": 13,
 "_unknown": 115,
 "_total": 131
 }
 }
}

Second, a index.json is generated which links to all the JSON files with statistics. This index.json file lists the relevant files for each day. If more than one file is listed, all files should be treated as partial and be combined.

Warning: listed files may also contain data for a different day, pay close attention to the date header.

The created index.json can then be consumed by the iod-stats-collector (to be published on stats.izzyondroid.org via iod-viz-stats, or directly by end-users who want per-source download stats. For more tips on understanding the output, see the section "Tips on understanding the output" further into the README.

Usage

This usage guide assumes the following setup, please replace locations as desired:

• Config file location: config/config.toml
• Server log file location: logs/log.txt
• Output directory for statistics: stats/

Before you get started, ensure you have a system running at least Python 3.11. On Debian 11 (bullseye), you could install python3-pytoml to use the script with an older Python version, but do note this is not officially supported.

First, set up a config.toml configuration file with a parser type. There are a few supported parser types:

Regex

The regex parser is the most common parser, used for example with Apache. To use the regex parser, set type under the [parser] section to regex.

To use the regex parser, you will need to write a regular expression containing at least the following matching groups:

• datetime
• path
• method
• statuscode
• useragent

An example config.toml for Apache is provided as config.toml.apache.example. We welcome example config files for different web servers.

Warning: It is important that the regular expression matches a complete line. Partial matches will be considered an error, to reduce the risk of a bad regular expression accidentally capturing private data. It is strongly recommended to filter your log files before giving it to the parser, for example using grep. For example, under default Apache rules, grep -E '\.apk .* 200 ' is quite a helpful filter.

Json

The JSON parser is used for web servers which directly log to JSON, like for example Caddy. To use the JSON parser, set type under the [parser] section to json.

By default, the JSON parser assumes all the fields are named the exact same in your JSON output, but this will rarely be the case. You can set json_field_* variables to explain what fields you need. For subfields, use ..

For example, if your JSON is {"request": "uri": "your_path"}, set json_field_path to request.uri.

If the field ends up as a list, the first value is taken (this should probably be improved).

One last thing to set up in config.toml is the date and time parsing.

By default, for timestamps (the datetime field) the Common Log Format is assumed, with a fallback to parsing as unix timestamps if this doesn't work. However, you can set datetime_format if you want to use another Python strptime format code string.

After setting up a config.toml file and having your web server logs ready, you can start generating your first .json file as such (this will write directly to stdout, redirect to to whereever you want to save the JSON file):

cat logs/log.txt | python3 iod_dl_parser.py --config config/config.toml > "stats/$(date +%s).json"

iod_dl_parser.py will return an exit code based on the result of the parsing:

• Exit code 0: No errors
• Exit code 1: An unexpected error occured
• Exit code 2: One of more lines did not match the defined regular expression (the non-matching lines are also printed to stderr).

Warning: iod-stats-builder has no way to detect duplicate data. It is your responsibility to not store the same download info in multiple files, as those statistics should be counted together.

After you have generated all the JSON files you want, make sure to put them all in the same root directory (you may, if wanted, use subdirectories to further sort them). You can then generate an index.json (into the root dir) as follows:

python3 iod_dl_index_generator.py --stats-dir stats/

Now, all that is left is to make the content of stats/ available for download. Whichever method you choose for publishing, it is important all relative links in index.json can be followed. If you store your files in git, make sure to allow raw access (supported by most forges) or set up some type of Pages setup.

Tips on understanding the output

The structure of the JSON files are explained on the top of this README. End users are recommended to view the final collected statistics on stats.izzyondroid.org.

However, here are some tips and tricks for understanding the output (aimed at developers).

VisiData

One option to manually browse the JSON files is using VisiData, which also lets you sort and more:

jq

You can run one of the following statements from its root (replacing me.hackerchick.catima with the name of the package you want statistics for):

# overall downloads for all stat files present
cat apk_downloads/*/*.json | jq -s 'map(.[][$appid]._total)|add' --arg appid me.hackerchick.catima
# day by day statistics over all stat files present
cat apk_downloads/*/*.json | jq -rs 'map(to_entries|map([.key,.value[$appid]._total]))[][]|select(.[1]!=null)|"\(.[0]): \(.[1])"' --arg appid me.hackerchick.catima

Both examples would need the jq package available on your machine. If you prefer some Python code, you can get some inspiration from this snippet.

License

Unless mentioned otherwise (check e.g. with the SPDX headers inside the files), all scripts in this repository are licensed as AGPL-3.0-or-later (see the LICENSE.txt):

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License version 3 as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

Funding

This project is funded through NGI Mobifree Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet program. Learn more at the NLnet project page.

NLnet foundation logo