Analysis of the Fons de Documentació del Medi Ambient

Charles H. Pence & Max Bautista Perpinyà
(analyses performed March-April, 2025, revised March, 2026)

In what follows, all code and text (CSV, HTML) files are available in this repository. All pickled Python data is available in a Figshare repository, with DOI X. (FIXME)

00. Document Analysis

The first step was to take raw HTML dumps of the Fons website, parse them into JSON files, and then parse these JSON files into the representations that we would analyze.

First (scripts not included here), the website's scraped HTML pages were converted into a general representation, available in fonsmeda.json. This includes:

• descriptors: the list of all "descriptors" that were used to tag different resources within the Fons.
• directory: the "directory" of organizations that was included within the Fons. These were given numerical identifiers, connected to the list of descriptors, and their name, abbreviation, and URL were saved. (This directory data was not analyzed further for the purposes of this project.)
• library: the actual referenced publications/documents that were available in the Fons. These are connected to the same list of descriptors (by ID), and were stored along with their bibliographic information and metadata, as it was presented in the pages of the Fons.

This file was then parsed into what was analyzed for this project, namely a representation of each document in library for topic modeling. This representation was created by combining the document's title, its "notes" field (if that field was longer than 32 characters), its keywords, and its descriptors (both of which were manually curated by the archivists at the Fons). These sentences were saved as an array in docs.json.

An example of such a document phrase:

Dispersio potencial dels contaminants atmosferics a Barcelona : estudi d'alguns parametres fonamentals a partir de les dades meteorologiques de l'observatori del Putget. Bibliografía p. 159-161. Biblio. meteorologica de Barcelona p.163. Mots clau: observatorio del putget, contaminantes atmosfericos, datos meteorologicos, contaminacion atmosferica

The script that created docs.json is 00-docs.py.

01. Embeddings

These documents were then embedded using the nomic-ai/nomic-embed-text-v2-moe embedding model, which was trained on a significant corpus of Spanish and Catalan text. Embeddings and a similarity matrix were saved as embeddings.pickle and similarity.pickle.

The script that created these embeddings is 01-embeddings.py.

02. Test Visualization

A quick, non-fine-tuned topic model and HTML visualization of this data was prepared in order to see whether or not it should be analyzed further. The script that produced it is 02-test-vis.py, although the resulting models and visualizations were not saved.

03. Final Models

Final topic models were produced using BERTopic, with 03-final-models.py. BERTopic parameters were:

• The dimensionality reduction model was UMAP, run with BERTopic's default parameters (n_neighbors=15, n_components=5).
• The vectorizer was amended by setting the stop words to a combined list of Spanish and Catalan stop words, taken from the analyzers in the Lucene project.
• Topics were labelled both with BERTopic's KEYBertInspired representation model, and by passing them to OpenAI's GPT-4o-mini.

A hierarchical topic model was then also prepared, using BERTopic's standard method. Finally, the years of documents (for which dates were available) were used to compute the distribution of these topics over time.

The creation of these models was peformed by 03-final-models.py.

04. Model Visualization

Topic details and visualizations were then produced in basic HTML and CSV formats, along with a dendrogram of the hierarchical model, and a graph of the topic distribution over time. These visualizations were created by 04-final-vis.py.

05. Merged Visualization

Further analysis revealed that the trends over time of the merged topics in the hierarchy would be interesting. These were visualized by extracting the tiers of the hierarchical visualization with 2 and 4 topics, merging the relevant topics in the model, re-labeling the resulting merged topics, and visualizing their distribution over time. This analysis was performed by 05-merged-vis.py.

06. Documents Over Time

A basic count of documents over time in the corpus was created by 06-histogram-time.py.

Credits

We would like to thank Alejandro de la Cueva and the rest of the team of the Fons de Documentació del Medi Ambient, who worked tirelessly for decades to make environmental knowledge useful and accessible. We would like to also thank José Ramón Bertomeu-Sánchez and Ximo Guillem-Llobat from the Institut Interuniversitari López Piñero, who put us in contact with former Fons activists and have also worked to keep the memory of the environmental movement alive and relevant.

License

Code in this repository is released under the GNU GPL v3 (see COPYING.code). The data and analysis results (including the parsed representations of the website of the Fons in docs.json and fonsmeda.json, as well as the topic models) are released under CC BY-NC-SA 4.0 (see COPYING.data).