salsah-export

Export scripts in Python for extracting metadata and data from Salsah, a virtual research environment developed by the Data & Service Center for the Humanities (DaSCH).

This GitHub repository was created in the context of the Participatory Knowledge Practices in Analogue and Digital Image Archives (PIA) project and specifically to extract content from three collections of the Swiss Society for Folklore Studies/Schweizerische Gesellschaft für Volkskunde (SSFS/SGV).

TOC

• Cloning and installation
• Usage
  • Export metadata from Salsah
  • Resolve Salsah metadata
  • Populate PIA database
• Background

Cloning and installation

Download ZIP or clone the repository

git clone Participatory-Image-Archives/salsah-export

To install the requirements:

pip3 install -r requirements.txt

Usage

Export metadata from Salsah

This step uses the salsah2xml.py script to export the data from the salsah database into a XML file. The script was originally provided by Lukas Rosenthaler of the Digital Humanities Lab, Basel.

Command

python3 salsah2xml.py --project {shortname} --shortcode {shortcode} --start {n} --nrows {n} --download --write-metadata --filter {shortname}:{resourcetype}={id} --request-image-type {jpg|jp2|png|tif} --restype {shortname}:{resourcetype} --batch-size {n} {USER} {PASSWORD} https://www.salsah.org

Main arguments

• project: shortname or ID of the project (sgv)
• shortcode: Knora shortcode (4102 for the SGV)
• --start: start at record with given number
• --nrows: amount of records
• --download: records are being downloaded
• --write-metadata: associated metadata are being written in an XML file
• --filter: supposedly a filter by resource type according to one of the Salsah ontologies
• --request-image-type: request a specified image format (JPEG, JPEG2000*, PNG or TIFF)
• --restype: download a specific resource type
• --batch-size: batch size to split download into (1000 by default)

NB: If JPEG2000 are requested from Salsah, only regular JPEG will be delivered.

The different resource types within the SGV ontology are:

• sgv:album
• sgv:collection
• sgv:image
• sgv:film
• sgv:sequence
• sgv:subject
• sgv:tonbildschau

The filter argument for collections can be used as follows: sgv:in_collection={ID}

The collections' ID can be found at the following URL on the Salsah API: https://data.dasch.swiss/api/resources?restype_id=67

Outputs

Below is a command example for downloading five images (JP2) and their associated metadata from the SGV_10 Familie Kreis collection:

python3 salsah2xml.py --project sgv --shortcode 4102 --start 3 --nrows 5 --download --write-metadata --filter sgv:in_collection=344305 --request-image-type jp2 --restype sgv:image {USER} {PASSWORD} https://www.salsah.org

After running the above command, the recors and their associated metadata will be downloaded in a sgv.dir folder (by default) structured in the following manner:

• assets - TBD
• images
• sgv.json (SGV Ontology)
• sgv.xml (metadata of the downloaded records) Screenshot 2021年04月28日 at 12 08 41

To get other metadata objects you can use calls like the following, in the case of Persons.

python3 salsah2xml.py --project sgv --shortcode 4102 --start 0 --nrows 2519 --write-metadata --batch-size 500 --restype sgv:persons {USER} {PASSWORD} https://www.salsah.org

Resolve Salsah metadata

This step works with resolve_properties.py to resolve the missing properties in the XML file and adds the capability to convert the resolved XML to a csv file. This script needs the exported metadata descriped in Export metadata from Salsah. See Unresolved Salsah metadata for more information.

Command

python3 resolve_properties.py --project sgv --csv True

Arguments

• --project: Reads the folder structure created by salsah2xml.py
• --csv True: Write the resolved and cleaned XML to a CSV file

The script generates a second XML file with the properties resolved by adding the following structure along the unresolved property id.

Outputs

Except for the svg:Image resource type where only PIA-related collections were extracted (SGV_10 and SGV_12), all of the SGV metadata have been exported using the above scripts. The available XML and CSV files can be found in the appropriate folders of the PIA Data Model GitHub repository: https://github.com/Participatory-Image-Archives/pia-data-model/tree/main/salsah-export

It this directory can also be found the Geonames and properties in JSON as well as a CSV which comprises all ASV places (asv_places.csv - SGV_05) and the current SGV ontology (sgv.json).

Populate PIA database

The purpose of populate_database.py is to populate a PostgeSQL database with the extracted metadata from Salsah. This script needs the exported metadata descriped in Export metadata from Salsah.

Setup database schema

Here are the main steps:

1. Create a PostgreSQL database
2. Import the PIA database model (SQL file)
3. Download the Salsah Export and put the unzipped folder in the root directory
4. Modify line 30 according to the created database and its credentials

After this you are able to run the population script.

Command

python3 populate_database.py

Background

Unresolved Salsah metadata

In the provided XML file, some properties are unresolved. For example

<resptr-prop name=":hasContent">
 <resptr>375013</resptr>
 <resptr>492859</resptr>
</resptr-prop>

The salsah.org API provides a JSON object for each property with their respective metadata via

https://www.salsah.org/api/resources/{ID}?reqtype=info

or respectively for GeoNames IDs:

https://www.salsah.org/api/geonames/{ID}?reqtype=info

See Salsah endpoints for more information.

Of interest are especially the value field firstproperty as well the user-given label restype_description. After successfully grabbing the metadata XML file, a second script will resolve the properties. Call the script on the same level as salsah2xml.py

The script also parses and resolves the other property types:

• text-prop
• list-prop

Salsah endpoints

As background, here are the different endpoints of Salsah.

It is to be noted that what is on https://www.salsah.org is equivalent (and will be later resolved) to what is on https://data.dasch.swiss.