ConGen: A graphical application for synthetic conservation planning data design

Overview of the main application window showing a smooth gradient raster layer

Full information

For full information, see the ConGen report or the ConGen presentation.

Cover of the ConGen report Title slide of the ConGen final presentation

Requirements

For python library requirements, see pyproject.toml. Additionally, the following system libraries and development headers are required:

• basic build tools (gcc, make etc.)
• gdal for building the Python gdal module
• gtk3 graphic libraries for the wxPython user interface

Usage

Locally installed (see below)

After installation, type

python -m congen

or (outdated)

congen-gui

to start the graphical interface. If installed system-wide or user-wide, this can be done from any terminal session. Otherwise, make sure to cd to the project folder and source bin/activate to activate the virtual environment first.

For additional usage instructions, see Getting Started.

Apptainer container

Using Apptainer, ConGen can be executed without installation. There is an Apptainer container definition available at congen.apptainer.def. A readily built container should be available at results/congen.apptainer. Just use

apptainer run <image name> <ConGen args>

to execute ConGen with the supplied arguments. Both the graphical interface and the command line versions are accessible using this way. Execute apptainer run <image name> -h to see the command line options for ConGen. Execute apptainer run <image name> without additional options to just use the ConGen UI.

Command-Line usage

Syntax:

--layer <Layer-Type> <Layer-Name> --parameter <Layer-Name>.<Parameter-Name> <Value>

Installation

Via pip for development

To install a local copy for development, first git clone this repository to your machine and cd into it. Then execute

python -m venv venv
source venv/bin/activate
pip install -e .

This will create a virtualenv in the repository folder and install the packages and all dependencies locally, without any changes to the system-wide python libraries. Furthermore, this will only install a reference to the files under src/ instead of copying them, so all changes made during development are immediately visible.

Via official pip package (outdated)

This project is available as an official pip package, and can be installed using

pip install congen

on any machine with access to a pip mirror, meeting the requirements above. It is recommended to not install packages to your machine's root, but rather to a user-local directory using

pip install --user congen

Using a virtualenv

Even better, use a virtualenv for this project:

mkdir congen-venv
python -m venv congen-venv
cd congen-venv
source bin/activate
pip install congen

Now the package and all dependencies will be installed inside the congen-venv-directory, and only be available in terminal sessions where source bin/activate has been typed from within the congen-venv directory (cd to it first). This way, no changes to your system python packages are required, and uninstallation is as easy as deleting the congen-venv directory.

Using your systems package manager (system-wide, outdated)

There is an arch linux package available for this project to install it system-wide. For this, download the PKGBUILD file or git clone the entire repository, cd to it, and then execute

makepkg -sri

This will install this project and all dependencies system-wide.

Known issues

Due to strange bugs in the deserialization of objects in the jsonpickle module, saving/loading project files does not work at the moment. Technical detail: Objects referencing other objects cause issues, instead of the reference to the correct object being restored, a completely different object is referenced, e.g. a parameter instead of a layer.

TODOs

• Fix WardClusteredPointLayer
• Command line interface
  • Might replace project saving / restoring by using command line strings instead
• Parallelisation / Multi-Threading for large datasets on slow CPUs
  • Dataset export
  • Random Walk generation (per starting point)
• Implement connectivity graphs
• Additional random models
• Additional import / export formats
  • Import and visualize csv data
    • Useful for checking datasets after export
• Benchmark different conservation planning software
• Improve UI / UX
  • Onboarding / Instructions
    • (?)-Button for most parameters / layers etc.
  • Project creation wizard with predefined templates?
  • Project template files
  • Tooltips for parameters
  • Multi-select layers, batch-adjust parameters
• Documentation, Manual
• Use logging framework
• Fix project loading/saving
  • Might be replaced by loading / saving command line strings?
• Automatically make currently selected layer the depending layer upon adding new layers
• Improve performance
  • Reduce CPU and memory footprint of graphs (own implementation, since we don't use much of iGraph anyway?)
    • Partially fixed by allowing to disable graph rendering, as the calculation itself is efficient enough
  • Option to pause rendering for instant adjustments
• Write tests
  • Check exported datasets
    • PU IDs match across graph, feature and quality data
    • PU coordinates match across data and PU IDs (i.e. no accidental transposing of the data etc.)
• Better separation of UI and logic
• Introduce different views, e.g. for features, quality data, intermediate layers etc., as it does not make sense to sum values of species occurrence data and quality data

Ideas

• Make layer_usage classes instead of a parameter, and dynamically generate layer classes that inherit the layer_usages
  • Add different menus to add feature or quality layers
  • Benefit: Less clicks for users, we don't need to add / remove parameters if layer usage changes
  • Con: More UI elements, more (dynamic) classes
• clustering of nodes in graph as probability.csv gets very big quickly
• use / compare to real geographical data (OpenStreetMap)
• Benchmark different conservation planning software
• Layer masks for Perlin Noise layers to create sparsity
• More sophisticated random walks:
  • Multiple random walks starting in same location to simulate multiple individuals
  • Negative random walk values: Individuals are repelled by that area, move away
• Focus on UI/UX improvement / best practices, testing
• Metrics to evaluate quality of generated data / similarity to real data
• Higher edge weights when crossing areas with lower feature values
• Animals should leave the city again upon entering: Transform city areas into gradients, values get more negative with increaisng distance from the city borders
• Limit starting points to habitats, e.g. fish in water
• Filter landscape types with connected area below threshold

Ideas after final presentation

• Split real data into parts for AI training
• Multiple terrain layers or one terrain layer with different components?
  • Have random walk probabilities depend on multiple filter layers with different proportion
  • Landscape borders? (e.g. ducks at the border of lake and grassland)
• Random Walk improvements
  • Random Walks as probability filter for random walks
  • Elevation data as probability filter for random walks
  • Have random walks stop at highways?
  • Stay on the same place for multiple iterations if probability is good?
  • Weather data as probability filter?
• Can we even compare conservation planning algorithms without knowing our data is good?
  • Isn't our data good enough already, given the low complexity of the conservation planning algorithms?

Master's thesis

Features

• Graph analyses
• Comparison of Conservation Planning Software
• Comparison of synthetic to real data (Metrics?)
  • AI?
• Obstacles (Highways) -> (Per-species) restriction layers?
• More / Better random models

Archievements

• Object-oriented design
  • All layer types are implemented as classes that inherit from common layer types, overriding common methods if needed
• wxPython GUI with Model-View-Controller Design
  • LayerListCtrl controller class manages both a wxPython list control and the underlying data structure
  • List view and underlying data structure are always in sync
  • On parameter change: Corresponding value in underlying data structure gets automatically updated, list view and rendering are automatically refreshed
• Custom implementation for parameters
  • Wrapping a single value (e.g. int, bool, string) with metadata (default, min, max values, parameter name and description, etc.)
  • Stored in a list for each layer type to distinguish parameters from other layer instance variables
  • Each layer type requires different parameters, extends the parameter list from inherited classes
  • Automatically render appropriate parameter controls for each layer type based on parameter list
  • Automatically render columns in layer list based on parameter list
  • Caching: Specify which parameters invalidate the calculated cache, only re-calculate layer data if relevant parameters change, don't recalculate for parameters that can be applied to existing data -> minifies computational overhead, especially for applications with many layers
  • Python magic so parameters of each layer can still be accessed via layer.parameterName despite being a) stored in a list variable instead of being class instance members themselves b) classes that wrap the actual value with additional metadata

Material

Ecology

• BC Marine Conservation Analysis

Procedural map generation

• Notes on Procedural Map Generation Techniques
• java - Procedural Map Generation - Stack Overflow
• Map Generation - Procedural Content Generation Wiki
• Procedural Map Generation with Godot — Part 1
• A Guide to Procedural Generation - GameDev Academy
• Procedural Map | Rust Wiki | Fandom
• Map generator - Minetest Wiki

Perlin Noise

• Noise and Turbulence - Ken's Academy Award -> Ken Perlin
• Lecture 14 Procedural Generation: Perlin Noise
• Perlin Noise - Scratchapixel
• Playing with Perlin Noise: Generating Realistic Archipelagos
• Perlin 2D Noise in python
• Exploring Perlin Noise in Python
• Perlin noise in python - Stack Overflow

Python Libraries

Not exhaustive

• EasyPerlinNoise
• pythonperlin
• perlin
• perlin-numpy
• perlin-cupy
• vec-noise
• noise
• noise-randomized
• perlin-noise
• nPerlinNoise
• pyfastnoisesimd
• pyperlin
• shades
• opensimplex
• noisemaker -> Olsen Noise(?)
• pyramage
• processing
• perlin.py