2
9
Fork
You've already forked cosmo
0
filter & transform OSM PBF data to geoparquet or geojson
  • Rust 99.5%
  • Just 0.5%
Martijn van Exel cf10e527d5
Some checks failed
CI / build (push) Has been cancelled
fix: flaky dense-cache free-space test racing with disk churn
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026年07月04日 00:06:49 +02:00
.forgejo/workflows dependencies for build env 2026年02月06日 12:00:12 -07:00
docs Document bbox filtering and CSV/MapRoulette output 2026年06月07日 19:27:37 -06:00
examples add MR example 2026年06月07日 20:28:34 -06:00
fixture touchdown 2025年12月25日 11:35:44 -07:00
src fix: flaky dense-cache free-space test racing with disk churn 2026年07月04日 00:06:49 +02:00
tests fix: flaky integration tests from temp filter filename collisions 2026年07月04日 00:04:44 +02:00
.gitignore Update .gitignore 2026年01月21日 08:14:51 -07:00
.mise.toml chore: add mise.toml pinning Rust 1.88.0 toolchain 2026年05月31日 10:19:20 -06:00
AGENTS.md docs: update docs and examples for relation support 2026年05月31日 10:51:59 -06:00
Cargo.lock metadata + bump osmpbf 2026年06月07日 20:26:30 -06:00
Cargo.toml metadata + bump osmpbf 2026年06月07日 20:26:30 -06:00
CHANGELOG.md chore: prepare 0.2.0 release 2026年07月04日 00:04:44 +02:00
CLAUDE.md Symlink from agents.md 2026年01月30日 09:11:12 -07:00
Justfile chore: add justfile 2026年06月09日 10:57:32 -06:00
LICENSE chore: explicit ISC license texy 2026年06月09日 11:30:22 -06:00
README.md doc: drop stale 0.2.0 branch note from README 2026年07月04日 00:05:24 +02:00

 ______ ______ ______ ___ __ __ ______ 
 /_____/\ /_____/\ /_____/\ /__//_//_/\ /_____/\ 
 \:::__\/ \:::_ \ \\::::_\/_\::\| \| \ \\:::_ \ \ 
 \:\ \ __\:\ \ \ \\:\/___/\\:. \ \\:\ \ \ \ 
 \:\ \/_/\\:\ \ \ \\_::._\:\\:.\-/\ \ \\:\ \ \ \ 
 \:\_\ \ \\:\_\ \ \ /____\:\\. \ \ \ \\:\_\ \ \ 
 \_____\/ \_____\/ \_____\/ \__\/ \__\/ \_____\/ 

Your favorite drink, now as a command line tool! This one won't give you a buzz, but it will convert your OSM PBF files into a bunch of other formats with really good filtering, which is arguably more useful.

Why not use...

  • osmium: Equally fast. Filtering is very basic. Only OSM output formats.
  • osm2pgsql: Arguably better filtering with Lua scripting, but output limited to PostgreSQL
  • gdal: swiss army knife of geospatial data manipulation, but not great at working with the OSM file format at scale.

There are many tools that fill the niche "I have OSM data but I want it in another format and transformed". Compare them and use what best fits your needs.

Quick Start

just commands are available for some common operations, see justfile for details.

Installation

cargo install --path .

Basic Usage

cosmo convert \
 --input city.osm.pbf \
 --filters config.yaml \
 --output output.geojson

Example Filter Configuration

Extract named restaurants and cafes as points:

table:name:poisfilter:'name & (amenity=restaurant|cafe)'geometry:node:trueway:centroid # Convert open ways to pointsclosed_way:centroidcolumns:- name:namesource:tag:name- name:cuisinesource:tag:cuisine- name:osm_idsource:meta:idtype:integer

See examples/ for more filter configurations and the Filter YAML Guide for a step by step guide and reference documentation.

Command Line Interface

Convert Command

Convert and filter an OSM PBF file.

cosmo convert [OPTIONS]

Required Options

Option Short Description
--input <PATH> -i Input OSM PBF file
--filters <PATH> -f Filter configuration (YAML)
--output <PATH> -o Output file (.geojson, .geojsonl, .parquet, .csv) or - for stdout in streaming formats (geojsonl, csv, maproulette)

Common Options

Option Short Description Default
--format <FORMAT> Override format detection: geojson, geojsonl, geoparquet/parquet, csv, maproulette Auto-detect from extension
--bbox <BBOX> Filter by bounding box: min_lon,min_lat,max_lon,max_lat None
--all-tags Include all OSM tags as JSON in addition to configured columns false
--verbose -v Enable detailed logging false
--jobs <N> -j Parallel processing threads All CPUs

--bbox filters completed output geometries by intersection. It does not clip geometries, produce smaller PBF extracts, or preserve topology outside the input; use osmium extract before Cosmo when you need those guarantees.

Node Cache Options

OSM ways reference nodes by ID, so Cosmo needs a way to look up node coordinates when building way geometries. The node cache is built during the first pass through the PBF file and used during the second pass.

Option Description Default
--node-cache-mode <MODE> Cache strategy: auto, sparse, dense, memory auto
--node-cache <PATH> Custom cache file path (dense mode only) Temp file in $TMPDIR if set, otherwise /var/tmp
--node-cache-max-nodes <N> Maximum node ID to allocate address space for (dense mode) 16,000,000,000

Cache Modes:

  • auto - Runs a node ID pre-scan, then selects memory for small inputs, sparse for sorted sparse ID distributions, or dense when direct indexing is smaller or sparse is unavailable. If you're converting a large PBF, set --node-cache-mode explicitly to avoid the initial node pre-scan.
  • sparse - Sorted array with binary search (O(log n)). Low memory, best for regional extracts with sparse node ID coverage. Requires sorted input - PBF files from OSM/GeoFabrik are sorted; use osmium sort if unsure
  • dense - Direct array indexing by node ID (O(1)). Memory-mapped file using OS sparse file support. Best for dense node ID coverage or unsorted inputs that do not fit memory. Auto mode sizes dense caches from the pre-scanned maximum node ID; explicit dense mode uses --node-cache-max-nodes. Use --node-cache to place it on a large disk
  • memory - In-memory HashMap (O(1)). No disk I/O but high RAM usage (~24+ bytes per node). Auto mode only selects it for inputs with at most 1,000,000 nodes; for planet processing you'd need 384GB+ RAM

Why these options? OSM node IDs are globally assigned, so even a small city extract can have IDs scattered across the global range. Auto mode uses real node count, max node ID, and sortedness instead of compressed file size: small inputs avoid disk with memory mode, sorted sparse extracts use compact sparse lookup, and dense ID ranges use O(1) mmap lookup.

Example: Extract POIs from a Region

cosmo convert \
 --input utah-latest.osm.pbf \
 --filters examples/pois_simple.yaml \
 --output pois.geojsonl \
 --bbox -112.1,40.5,-111.8,40.8 \
 --node-cache-mode sparse \
 --verbose

Validate Command

Validate a filter configuration without processing data:

cosmo validate <config.yaml> [--verbose]

Environment Variables

All options support environment variables with the COSMO_ prefix:

export COSMO_INPUT=city.osm.pbf
export COSMO_FILTERS=config.yaml
export COSMO_OUTPUT=output.geojsonl
cosmo convert

Filter Configuration

Filters are defined in YAML and consist of:

  1. Filter DSL - Which features to extract
  2. Geometry configuration - How to handle nodes/ways/relations
  3. Columns - Which tags/metadata to include

Filter DSL Quick Reference

Syntax Description Example
key Tag exists highway
key=value Exact match highway=primary
key=v1|v2 Match any value amenity=cafe|restaurant
expr1 & expr2 Logical AND highway & name
expr1 | expr2 Logical OR amenity | shop
key=*pattern* Glob matching highway=*_link
key>=value Numeric comparison lanes>=2
meta:version >= 2, meta:timestamp >= "2024年01月01日" OSM metadata comparison Matches by version, timestamp, user, uid, changeset, visibility, id, or element type

See the Filter YAML Guide for complete syntax and examples

Column Sources

Extract data from OSM features:

Source Description Example
tag:key OSM tag value source: tag:name
meta:field Metadata field (id, type, version, timestamp, uid, user, changeset, visible) source: meta:id
tags All tags as JSON source: tags (requires type: json)
meta All metadata as JSON source: meta (requires type: json)
refs Way node IDs as JSON array source: refs (requires type: json)
mapping:name Apply categorization mapping source: mapping:poi_class
const:value Literal value typed by the column's type source: const:buildings-v1
expr:CEL Compute with CEL expression source: 'expr:has(tags.name_en) ? tags.name_en : tags.name'

Column Types: string (default), integer, float, json

See the Filter YAML Guide for details and advanced features

Spatial Filtering

In addition to tag-based filtering, the --bbox min_lon,min_lat,max_lon,max_lat option restricts output to features whose resolved geometry intersects a bounding box (see Common Options). It filters by intersection only — it does not clip geometries or produce a smaller PBF; use osmium extract first when you need a topology-preserving cut.

cosmo convert -i utah-latest.osm.pbf -f config.yaml -o pois.geojsonl \
 --bbox -112.1,40.5,-111.8,40.8

Output Formats

GeoJSON

Complete FeatureCollection in a single file. Best for small extracts and web mapping.

GeoJSONL (Newline-Delimited GeoJSON)

One Feature per line. Supports streaming to stdout (--output -) for piping to other tools:

cosmo convert -i data.osm.pbf -f config.yaml -o - | tippecanoe

GeoParquet

Columnar format with spatial metadata. Efficient for large datasets and analytics.

GeoParquet writes only declared columns plus geometry by default. When --all-tags is enabled, it also writes a properties JSON column containing the extra tag payload.

Note: GeoParquet output is not spatially sorted. For best performance, sort with DuckDB's Hilbert function. See CloudNativeGeo best practices.

CSV

CSV with declared columns plus a WKT geometry column. Supports streaming to stdout (--output -).

MapRoulette

MapRoulette is a platform for creating and managing OSM mapping tasks. Cosmo's maproulette format writes RFC 7464 records: each line starts with the record-separator byte and contains a FeatureCollection with one task feature, ready for MapRoulette's line-by-line GeoJSON challenge upload.

Documentation

License

ISC