xmloot

Transform arbitrary XML data into a nice, relational database

This is a tool to transform XML data into a relational database. It can be configured to read arbitrary data from XML files and, from that, it builds a relational database.

Usage

To learn more about xmloot's command line interface, use:

$ python -m xmloot --help
usage: xmloot [-h] [--globbed] [--encoding ENCODING] [--workers WORKERS]
 [--report REPORT] [--db DB] [--schema SCHEMA] [--create-schema]
 [--overwrite] [--no-validate] [--progress] [--log-level LOG_LEVEL]
 transformation file [file ...]
positional arguments:
 transformation path to a transformation YAML file containing the
 transformations to apply
 file files to apply transformations on
options:
 -h, --help show this help message and exit
 --globbed whether to evaluate file parameter as globs
 --encoding, -e ENCODING
 encoding of the XML files (default: utf-8)
 --workers WORKERS The number of workers to use. The default is to not use
 multiprocessing. Set to 0 to choose by number of
 processors.
 --report REPORT .jsonl file to write report on inconsistencies to
 --no-validate do not validate the config file syntax
 --progress Show progress bar on stderr
 --log-level LOG_LEVEL
 Set the log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
database:
 --db DB database connection string
 --schema SCHEMA database schema to use
 --create-schema whether to create the database schema
 --overwrite delete and re-create the schema beforehand if it exists

Tip: Use the uv package manager to automatically download dependencies.

In its simplest form, the command takes the following arguments:

• A configuration file, written in YAML, which describes the parts of the XML that should be put into a database. For a description of the configuration file, see Configuration.
• A list of XML files to transform

Configuration

The configuration file is written in YAML. A schema is provided at xmloot/resources/xmloot-config-schema.json. See qualitaetsberichte-database for a usage example.

A configuration file defines the tables of the resulting database. For each table attribute, an XPATH is specified to retrieve its values. Additionally, tables can be nested. Nesting tables makes it possible to define one-to-many relations, with child tables inheriting their parent table's primary key as a foreign key.

License

Copyright © 2024-2025 Flora Rädiker, Philip Maus

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.