POI Complete

POI Complete is a Python project whose main goal is to help improving the data coverage and quality of OpenStreetMap (OSM) point of interests (POIs). From OSM POIs, it crawls associated websites, uses ML/LLM to extract opening hours information, and provides a REST API for reviewing predictions.

Features

• Extract POI data from OSM PBF files (imported via osm2pgsql)
• Crawl associated websites to extract content (save as HTML and markdown)
• Use ML/LLM (pydantic-ai with OpenRouter) to extract opening hours
• Store results in PostgreSQL
• REST API for reviewing predictions
• Vue.js frontend for annotation workflow

Tech Stack

• Python: 3.12+
• Database: PostgreSQL with Peewee ORM
• Web Framework: FastAPI
• Crawler: Crawlee
• ML/LLM: pydantic-ai with OpenRouter
• CLI: Typer
• Frontend: Vue.js 3 with TypeScript, Pinia, Vite, vue-i18n

Project Structure

src/poi_complete/
├── __main__.py # CLI entry point with all commands
├── api.py # FastAPI REST endpoints
├── config.py # Settings via pydantic-settings
├── models.py # Peewee database models
├── types.py # Enums and dataclasses
├── utils.py # Utility functions
├── parse.py # HTML parsing utilities
├── crawl.py # Web crawler implementation
└── predictions/
 ├── importer.py # Prediction/insight import logic
 └── opening_hours/ # Opening hours extraction
 ├── tasks.py
 ├── schemas.py
 └── utils.py

Database Tables

Installation

Prerequisites

• Python 3.12+
• PostgreSQL
• Node.js (for frontend)

Backend Setup

uv is recommended for managing the Python environment.

1. Create the virtualenv and install dependencies:

   uv sync
   

2. Configure environment variables

This is optional, as we provide sensible defaults for a local setup.

3. Create database tables:

   poi-complete create-tables
   

This create all tables except pois, which is created by osm2pgsql when importing OSM data (see below).

OSM Data import

POI data is imported from OSM PBF files using osm2pgsql with a custom Lua configuration.

The import process uses the Lua configuration at config/pois.lua, which:

1. Imports POIs from OSM data with keys: amenity, shop, tourism, leisure, office, craft, historic, man_made, aeroway, healthcare
2. Extracts common attributes: name, opening_hours, website, phone, operator, addr_city
3. Stores all tags in a JSONB column for flexibility

Prerequisites:

• An OSM PBF file (e.g., from Geofabrik)

Import command:

docker compose run --rm osm2pgsql \
 --host postgis \
 --database postgres \
 --user postgres \
 --style /config/pois.lua \
 -O flex \
 --slim \
 /pbf/your-region.osm.pbf

The import creates the pois table with the following schema:

Note: The pois table is created automatically by osm2pgsql during import.

Partial update

To update our local database to be synchronized with OSM, we use the osm2pgsql-replication CLI.

We first need to run the initialization:

docker compose run --rm osm2pgsql osm2pgsql-replication \
 init \
 --host postgis \
 --database postgres \
 --user postgres

Then, on a regular basis (e.g., daily), we can run the update command:

docker compose run --rm osm2pgsql osm2pgsql-replication \
 update \
 --host postgis \
 --database postgres \
 --user postgres \
 --verbose -- \
 --style /config/pois.lua \
 -O flex

Frontend Setup

cd front
npm install

Configuration

Environment variables:

Usage

CLI Commands

# Create database tables
poi-complete create-tables
# Run crawler on all POIs
poi-complete crawl --batch-size 100 --max-concurrency 8
# Run crawler with filters and advanced options
poi-complete crawl \
 --filter-category amenity \
 --filter-type restaurant \
 --max-retries 3 \
 --max-crawl-depth 1 \
 --enable-poi-rate-limit
# Crawl specific POI
poi-complete crawl-poi n123456789
# Extract opening hours from crawl result
poi-complete extract-opening-hour <crawl_result_id>
# Generate insights from all crawl results
poi-complete generate-opening-hours-insights --limit 1000
# Generate insights for POIs missing OSM data only
poi-complete generate-opening-hours-insights --missing-only
# Refresh insights for all POIs
poi-complete refresh-insights
# Drop a table
poi-complete drop-table <table_name>

Crawl Command Options

API Endpoints

# Get insights (with optional filters)
GET /api/v1/insights?annotated=false&types=opening_hours&limit=100&missing=true
# Parameters:
# - annotated: Filter by annotation status (true/false/null)
# - types: Filter by insight types (comma-separated)
# - limit: Number of results (1-250, default: 100)
# - missing: If true, only insights adding new OSM data; if false, only updates
# Annotate an insight (accept=1, reject=0)
POST /api/v1/insights/{insight_id}/annotate?annotation=1

Running the Backend API

uvicorn poi_complete.api:app --reload

A make command is also available for convenience:

make up-server

The API runs on http://localhost:8000.

Running the Frontend

cd front
npm run dev

A make command is also available for convenience:

make up-front

The frontend runs on http://localhost:5173 and proxies API requests to the backend.

Frontend Features

• Displays one insight at a time for review
• Shows POI name, website URL, current vs predicted opening hours
• Shows raw text used for prediction and LLM comments
• Indicates whether insight adds new data or updates existing OSM data
• Accept/Reject buttons to annotate insights
• Progress bar showing position in the batch
• Auto-loads more insights when running out
• Filter insights by missing OSM data status
• Open in JOSM button to directly edit in OSM editor

Code Quality

Linting/Formatting

ruff check .
ruff format .
isort src/ --profile black

Type Checking

pyright src/

Adding a New Prediction Type

1. Add type to types.py (PredictionType enum)
2. Create prediction extraction logic in predictions/
3. Create importer class in predictions/importer.py (subclass InsightImporter)
4. Add importer to IMPORTERS list
5. Add CLI command if needed in __main__.py

Adding Translations

1. Add translation keys to front/src/i18n/en.json
2. Add corresponding translations to front/src/i18n/fr.json
3. Use in components with t('key.path')