mangrove-osm-coder

A tool that maps reviews from https://mangrove.reviews to places on OpenStreetMap (OSM).

Setup

Install:

• docker
• just

Quick Start

git clone https://codeberg.org/mmakowski/mangrove-osm-coder.git
cd mangrove-osm-coder/run
just regions_file=regions.example.txt date=260101 import download-mangrove osm-code

This will populate the PostGIS database with three small regions as of 2026年01月01日, download the latest mangrove.reviews data dump and run the OSM coder tool. Due to the regions being small and somewhat off the beaten track, expect very few found locations. Still, it illustrates the workflow without overwhelming the OpenStreetMap data providers and your machine. Read the Configuration section to see how to set up the tool with a more comprehensive and more recent data set.

Output

The osm-code goal writes files in the output directory (run/out by default).

reviews.json

The reviews.json maps of OSM place (element) ids to the list of reviews, and is the primary output of the tool intended as an input to map generation pipelines.

Format

{
 <osm-element-id>: {
 "average_rating": <rating>,
 "reviews": [
 {
 "iat": <timestamp>,
 "rating": <rating>,
 "opinion": <string|null>,
 "author": <string|null>
 },
 ...
 ]
 },
 ...
}

where:

• <osm-element-id> is the OpenStreetMap element id in the format <osm-element-type>/<number>;
• <osm-element-type> is one of node, way, relation;
• <number> is a positive integer;
• <rating> is a number between 0 and 100;
• <timestamp> is a unix timestamp in seconds;
• <string|null> is a string or null.

The reviews list is sorted by iat in descending order (newest first).

Example

{
 "node/11015508055": {
 "average_rating": 75,
 "reviews": [
 {
 "iat": 1688325823,
 "rating": 75,
 "opinion": "For fast food quite long wait, but good taste and big portions. Also friendly staff",
 "author": "Karthoo"
 }
 ]
 },
 "way/980976422": {
 "average_rating": 100,
 "reviews": [
 {
 "iat": 1631170633,
 "rating": 100,
 "opinion": "Sehr sch\u00f6nes, neues Haus, sehr geschmackvoll und gem\u00fctlich eingerichtete Apartements! Wir haben die Gastfreundschaft von Sabine und Benni sehr genossen und kommen gern wieder!",
 "author": "elias"
 },
 {
 "iat": 1631170008,
 "rating": 100,
 "opinion": "Wundersch\u00f6ne Apartments, sehr nette Gastgeberfamilie! Alles Top, danke!",
 "author": null
 }
 ]
 }
}

Miscellaneous

The following files are also written to the output directory:

• review_to_osm.tsv: a map of review signature to OpenStreetMap place (element) id.
• fuzzy_name_matches.tsv: lists place names that did not match exactly but were still considered to match. Useful for tuning the name similarity threshold configuration parameter.
• not_found.tsv: lists metadata of reviews that were not successfully resolved to a place on the OpenStreetMap. Useful for identifying problems with the resolution logic.
• stats.json: a summary of the data processing statistics.

Configuration

Regions

The regions to process are specified in the regions file. By default, the commands expect the regions file to be named regions.txt and reside in the run directory. A sample file with format description is provided as run/regions.example.txt. It is recommended that you copy it to run/regions.txt and customise as you need. As an alternative, the regions file can be specified as a variable for download-osm and import goals, as we did in Quick Start.

Depending on the region selection, you will need between 1 GB (a few small regions) and over 100 GB (the entire planet) of free disk space. You will also need plenty of RAM for large imports: at least 64 GB for the planet file.

Date

All OpenStreetMap data used for resolution should be as of the same point in time. That point is specified as date=yymmdd variable for download-osm and import goals. For example:

just date=260330 download-osm

will download the OpenStreetMap data for regions specified in regions.txt as of 2026年03月30日.

Note that historical data is only available for certain dates. If using planet region, check out https://planet.openstreetmap.org/pbf/ for available dates. In the case of smaller regions, the dates are shown in the relevant sub-pages of https://download.geofabrik.de.

Currently, the date only applies to OpenStreetMap data. download-mangrove goal always downloads the latest available mangrove.reviews data.

Directories

The directories are configurable with environment variables. It is recommended to specify them in run/.env file. A sample run/.env.example file is provided for convenience.

Other settings

The tool is configured with environment variables. See run/.env.example file for a complete list and descriptions.

Data preparation

In run directory, run:

just date=260101 download-osm download-mangrove import

This will:

1. download OpenStreetMap data files; this can take a few hours, depending on the chosen region set and the speed of your connection.
2. download Mangrove reviews dump; this should be quick.
3. import OpenStreetMap data into PostGIS; for the entire planet this might take a few hours.

Note: PostGIS data must all be as of the same date. If you want to change the date, you will need to start from scratch:

just reset

This will delete the entire PostGIS database so that you can import a different date. The tool does not currently support incremental imports with OSM change files. Attempts have been made to make that work, but it turned out that osm2pgsql requires a very long time to build an appendable (slim) version of the database, so it is more practical to reload from scratch.

Running

In run directory run:

just osm-code

Once this completes you should see tsv files in run/out directory.

Browsing the Database

Once PostGIS is running, you can connect to it from PostgreSQL tools by configuring the connection as follows:

Development

First, follow the Setup and Quick Start steps to initialise the PostGIS instance.

IDE setup

Update the run configuration for mangrove-osm-coder to set the following environment variables:

MANGROVE_FILE=run/cache/mangrove.reviews/dump.json
OUT_DIR=run/out
POSTGIS_HOST=localhost

Then start the PostGIS instance by running, in the run directory:

just start

mangrove-osm-coder started from the IDE will then be able to connect to the PostGIS instance running in the docker container.

Run using just

To run the current development version of the Docker container, change run/.env to have:

CODER_VERSION=dev

then, at the top level:

just package-dev

to build a container from the current source.

Release

just release

Licence

GPLv3